Some thread safe and unsafe thread inside the collection?
Safe:
Vector
HashTable
StringBuffer
Not thread safe:
ArrayList:
LinkedList:
HashMap:
HashSet:
TreeMap:
TreeSet:
StringBulider:
The difference between mybaits and hibernate
hibernate: is a standard ORM framework (object relational mapping). The entry threshold is relatively high, there is no need to write SQL, and SQL statements are automatically generated, so it is difficult to optimize and modify SQL statements.
Application scenario: It is suitable for small and medium-sized projects with few changes in requirements, such as: background management system, erp, orm, oa, etc.
Mybatis: Focus on SQL itself, requiring programmers to write SQL statements by themselves, and it is more convenient to modify and optimize SQL. Mybatis is an incomplete ORM framework. Although programmers write sql themselves, mybatis can also implement mapping (input mapping, output mapping)
Advantages
of Hibernate Hibernate's DAO layer development is simpler than MyBatis, and Mybatis needs to maintain SQL and result mapping.
Hibernate's maintenance and caching of objects is better than MyBatis, and it is more convenient to maintain objects that are added, deleted, and checked.
Hibernate database portability is very good, MyBatis database portability is not good, different databases need to write different SQL.
Hibernate has a better second-level cache mechanism and can use third-party caches. The caching mechanism provided by MyBatis itself is not good.
Mybatis advantages
MyBatis can perform more detailed SQL optimization and can reduce query fields.
MyBatis is easy to master, while Hibernate has a higher threshold.
The difference between interceptors and filters?
①Interceptors are based on java's reflection mechanism, while filters are based on function callbacks.
② The interceptor does not depend on the servlet container, and the filter depends on the servlet container.
③Interceptors can only work on action requests, while filters can work on almost all requests.
④Interceptors can access objects in the action context and value stack, while filters cannot.
⑤ In the life cycle of action, the interceptor can be called multiple times, while the filter can only be called once when the container is initialized.
⑥ The interceptor can obtain each bean in the IOC container, but the filter cannot. This is very important. Injecting a service into the interceptor can call the business logic.
The difference between session and cookie
session is stored on the server side, and cookie is stored on the client side, so the security of session is higher than cookie.
The information in the obtained session is obtained through the sessionId stored in the session cookie
Because the session is stored in the server, the continuous increase of the content in the session will increase the burden on the server. We will put some important things in the session, and put the less important things in the client cookie.
Cookies are divided into two categories. , One is session cookie and persistent cookie. Their life cycle is the same as that of the browser. When the browser closes the session cookie, it will disappear, and the persistence will be stored in the client's hard disk.
When the browser is closed, the callback cookie will disappear, so our session will also disappear. Under what circumstances is the session lost, that is, when the server is closed, or the session expires (30 minutes by default).
Scala?
Scala is a multi-paradigm programming language designed to integrate various features of object-oriented programming and functional programming.
Scala runs on the Java Virtual Machine and is compatible with existing Java programs.
Scala source code is compiled to Java bytecode, so it can run on the JVM and call existing Java class libraries.
Kafka common sense
Fault tolerance, good scalability and advantageous performance, the use of Kafka is mainly used to process logs
The whole MapReduce process of MapReduce
is roughly divided into Map–>Shuffle (sort)–>Combine (combination)–>Reduce
The process of MapReduce is first started by the client submitting a task, which is mainly submitted through JobClient.runJob(JobConf) Implemented by static function
Read the content of the data file, parse the content into key-value pairs, call the map function once for each key-value pair, write the processing logic of the mapping function, build the input key-value pair and output a new key-value pair
. Partition, sort, and group value
pairs Copy
the output of multiple maps to different reduce nodes through the network according to different partitions Sort, merge, and write reduce function processing logic to convert the received data Create a new key-value pair
and finally save the data output by the reduce node to HDFS
Let's talk about one of the three core components of Yarn
Hadoop;
it is responsible for resource scheduling during massive data operations, and the roles in the cluster are mainly ResourceManager / NodeManager
◆MapReduce execution process on YARN: The
client executes JobClient —> requests the task id to ResourceManager —> Create a task ID —> return the task ID to the client —> the client starts uploading the task Jar package and obtains the original data of HDFS —> requesting Yarn to execute the task —> ResourceManager starts initializing the task —> ResourceManager starts assigning Tasks –> HDFS gets tasks and data from NodeManager
YARN can deploy a variety of computing frameworks (such as offline processing MapReduce, online processing Storm, iterative computing framework Spark, stream processing framework S4, etc.) to a public cluster and share the resources of the cluster
ResourceManager(RM):
Responsible for unified management and scheduling of resources on each NM. Allocate the AM to an idle Container to run and monitor its running status. A corresponding idle Container is allocated to the resource request applied by the AM. Mainly consists of two components: scheduler and application manager
NodeManager (NM) :
NM is the resource and task manager on each node. It will periodically report the resource usage on this node and the running status of each Container to the RM; at the same time, it will receive and process Container start/stop requests from the AM.
ApplicationMaster (AM):
The applications submitted by users include an AM, which is responsible for application monitoring, tracking application execution status, restarting failed tasks, and so on.
Container:
Container is a resource abstraction in YARN, which encapsulates multi-dimensional resources on a node, such as memory, CPU, disk, network, etc. When AM applies for resources to RM, the resource returned by RM to AM is Container. indicated. YARN assigns a Container to each task and the task can only use the resources described in the Container.
Application scenarios of Zookeeper
Unified naming service, unified configuration management, unified cluster management, dynamic online and
offline , soft load balancing
HBase storage principle
As mentioned above, hbase uses MemStore and StoreFile to store updates to tables.
Write operation:
Step 1: The client sends a data write request to the HRegionServer through the scheduling of Zookeeper, and writes the data in the HRegion.
Step 2: Data is written to the MemStore of the HRegion until the MemStore reaches a preset threshold.
Step 3: The data in the MemStore is flushed into a StoreFile.
Step 4: With the continuous increase of StoreFile files, when the number of StoreFile files increases to a certain threshold, the Compact merge operation is triggered, and multiple StoreFiles are merged into one StoreFile, and versions are merged and data deleted at the same time.
Step 5: StoreFiles gradually form larger and larger StoreFiles through continuous Compact merging operations.
Step 6: The last file StoreFile will be flushed and a storage file HFile will be generated accordingly. The continuous writing of the data volume will cause frequent flushing of the memstore. Each flush will generate an HFile, so that the number of HFile files on the underlying storage device will be reduced. There will be more and more. (Disk) It is best to write to HDFS through the HDFS interface
Kafka writing process
1) The producer first finds the leader of the partition from the "/brokers/.../state" node of zookeeper
2) The producer sends the message to the leader
3) The leader writes the message to the local log
4) The followers pull the message from the leader , send ACK to the leader after writing to the local log
5) After the leader receives the ACK of all replications in the ISR, it increases HW (high watermark, the offset of the last commit) and sends ACK to the producer