The first two articles talked about the value and construction ideas of the data warehouse. Here, a data warehouse model is implemented based on the actual business system data. The most recent project is the construction of the data warehouse of the bidding system. The business logic involved is more complicated and there are many participants. The amount of data is not large. After the data warehouse is built, it is mainly to support the process supervision level of bidding and procurement;
According to the construction idea of the data warehouse, the order is conceptual model -> logical model -> physical model. The most important and complex is the design of the conceptual model, which needs to be combined with the business, and the fact table and dimension table are designed according to the business characteristics , Top-level data summary table;
1. Conceptual model design
The conceptual model needs to be combined with the ER relationship model of the production system to sort out business logic. The current production transaction system uses the ORACLE database, which divides the data into multiple databases: business database (including the process of bidding and purchasing projects), subject + organization database (bidder, Bidders, bid evaluation experts, agencies), financial libraries (bidding fees, platform service fees, bidding deposits, CA handling fees, etc.), the project table is a bidding process table, which will record the bidding process Data related to bidding, bid opening, bid evaluation, and bidding:
Bidding: The bidding process is initiated by the bidder. The bidder entrusts the bidding process to the agency . The agency will issue a bidding announcement . The bidder will generate data during the registration and response stage, and the bidding deposit shall be paid after the response;
Bidding: The bidder pays the bidding fee to the agency and downloads the bidding documents. The bidder needs to respond before opening the bid and pay the bid bond; after the bidding documents are sold and the bidder purchases the bid, if the bidder questions the bidding documents or the bidder wants to modify the bidding Documents, at this time, a clarification announcement shall be issued within the specified time.
Bid opening: Bid opening is generally carried out offline. The agency calls the bidders to the bid opening room and publicly announces the bidder’s quotation, construction period, quality, project manager and other bidders’ substantive requirements. At this stage, the bidding documents are opened , Decrypt electronic bidding documents;
Bid evaluation: Bid evaluation is generally carried out offline. The agency summons supervisors, bidders, and experts to the bid evaluation room, and the experts score bidders’ qualifications and bids, which are divided into technical, commercial, and quotation points;
Bid determination: After the experts comprehensively score the bidders, they will make an overall ranking. The first ranked candidate is the winning candidate. After the bid evaluation is completed, a pre-winning announcement is required. The top three are announced. The announcement period is subject to social supervision. Questions and queries need to be clarified by the agency/tenderee, and the clarification is accompanied by a clarification announcement. If the query becomes effective, the bid may be invalidated (the bid evaluation cost is high, and the bid is generally not abolished);
Contract: After the pre-winning bid is released, the questioning period has no effect on the pre-winning candidate. xxx days after the pre-winning bid is released, the tenderee needs to sign a contract with the winning candidate, and the tenderee needs to return the deposit of other units that have not won the bid;
After sorting out the entire process and understanding the business, customers pay more attention to the supervision and early warning of the process, and sort out some supervision dimensions based on this:
2. Logical model design
The logical model adopts the dimensional modeling model mentioned in the previous blog post, the snowflake model, the project ID, bidder ID, bidder ID, agency ID, and expert ID are the entire process of bidding, investment, opening, evaluation, and bidding. The main participants, the data extraction tool uses kettle:
Data table naming convention: tb_model level_subject domain_business domain_summarization granularity
Kettle naming convention: kt_model level_topic domain_business domain_summarization granularity
3. Physical model design
According to the physical model design mentioned in the previous blog, build the layered model of ODS-->DWD-->DWS-->ADS. Here, ODS only extracts the source data in the oracle library without any cleaning or changes. The DWD layer Start to do data cleaning and data engineering, DWS for light summary, ADS provides higher-level summary for application query; take the summary dimension of project and supplier as an example, the project process is the main body of model design, and the supplier is similar to the dimension table Data, the combination of the two can get some summary dimensions related to the investment/winning of the business (such as the ranking of the winning rate, the statistics about the registered amount of the bidders of a certain project, the statistics about the participation of a certain bidder in the bidding, etc.):
In the project process table (calibration process), design the number of the tenderee, and the statistical items of the calibration process will obtain the results from the summary dimensions of this type of ADS:
