background
Digital certificate and a private key necessary for application CSR data, the public and private key pair generation, such as in particular the high intensity secret key 2048, an average single consuming close 300ms, and the change with time of generating a random number and greater fluctuation ;
KeyPairGenerator generator = KeyPairGenerator.getInstance("RSA");
generator.initialize(2048, new SecureRandom());
long startTime = System.currentTimeMillis();
for(int i = 0; i < 100; i++) {
generator.generateKeyPair();
}
System.out.println("cost: " + (System.currentTimeMillis() - startTime));
复制代码In order to ensure a rapid response to the secret key must be pre-generated and stored in the secret key pool, you can read the digital certificate in the application directly from the secret key pool;
In early systems, due to the small scale of business and stable business volume, in order to save the cost of deployment, key generation and certificate application running in the same service right, a simple model of producers and consumers to achieve:
- Continuous thread pool to generate a secret key, the core thread pool threads = maximum number of threads = (Runtime.getRuntime () availableProcessors () - 1.);
- LinkedBlockingQueue saved using the generated secret key pair and set the queue limit 2000;
problem
One day, a customer needs the Internet for online banking activities, known as customers and users to create a digital certificate application process is more time-consuming, so the customers to choose in advance batch create users, and then online activities. The results were predictable:
- Local service certificate request secret key pool is exhausted, the cache breakdown;
- Business synchronous trigger generated secret key pair, intensified competition for CPU resources, generate a secret key pool more slowly, the user creates a lot of overtime;
- Service is not unified network exit, there is a limit IP whitelist CA center, expansion can not be done quickly;
- Even if the expansion is completed, the pool is empty because the secret key, secret key generating synchronization service remains, the general effect;
Finally, only resorted killer: restrict customer traffic and communication with customers say the system was limiting ......
optimization
Typical problems following the original business model:
- To generate a secret key belonging to the background computation-intensive business, regardless of the user business, but business there is competition for CPU resources, can lead to unstable performance certificate application business;
- Secret key pair generation capacity and operational capacity of the coupling, can not achieve the rapid expansion of individual capacity;
- Each Queue data can not be quickly unified statistics, can not be quickly and easily monitor the number of objects in the cache keys in the pool;
- After restarting the service has generated secret key to disappear, causing the system does not allow rapid service restarts;
In response to these problems, optimize the following ideas:
- Redis introduced as a centralized cache buffer pool by decoupling the user service and back-office tools, and keys to solve the problem of lost restart pool;
- List structure using the cache memory redis secret key pair, the secret key of the maximum limit of 10W;
- An alarm when the number of elements in List redis be monitored, less than 35% of the total number, and thus can implement the service cluster dynamic automatic expansion (Write toss);
- Condition control using multiple queues reetrantLock and production control of the characteristics of producers;
- Key generation service thread pool generated secret key pair, when the total number redis pool is full, the thread enters await state, waiting to be awakened;
- Key generation using the service based on the number of objects in the pool Spring Schedule fixed speed polling interval of 5s, 75% less than half of the fixed thread pool threads wake generated secret key; wake up all thread pool threads generate secret keys to less than 50%.
Simplify implementation
Considering the number of threads is much less than the number of objects in the cache, the control 10W is only an approximation, a few more than a few less will not cause much impact on redis itself in the current business scenario
Secret key pair generation worker
/**
* 秘钥对生成执行线程池
*/
public final class KeyPairExecutors {
// Redis Client 桩类
private final RedisCacheDAO cacheDAO;
// 锁
private final ReentrantLock lock;
// 与线程数量相当的condition
private final List<Condition> workerConditions;
public KeyPairExecutors(ReentrantLock lock, List<Condition> workerConditions) {
this.cacheDAO = new RedisCacheDAO();
this.workerConditions = workerConditions;
this.lock = lock;
}
/**
* 工作线程构造方法
*/
public void start() {
int coreNum = workerConditions.size();
ThreadPoolExecutor poolExecutor = new ThreadPoolExecutor(coreNum, coreNum, 120, TimeUnit.SECONDS,
new SynchronousQueue<>(),
new KeyPairGeneratorThreadFactory("keypair_gen_", workerConditions.size()),
new ThreadPoolExecutor.DiscardOldestPolicy());
Stream.iterate(0, n -> n + 1).limit(workerConditions.size())
.forEach( i -> poolExecutor.submit(new KeyPairRunable(cacheDAO, lock, workerConditions.get(i))));
}
class KeyPairRunable implements Runnable {
private final RedisCacheDAO cacheDAO;
private final ReentrantLock lock;
private final Condition condition;
public KeyPairRunable(RedisCacheDAO cacheDAO, ReentrantLock lock, Condition condition) {
this.cacheDAO = cacheDAO;
this.lock = lock;
this.condition = condition;
}
@Override
public void run() {
while(true) {
String keyBytes = genKeyPair();
try {
int currentSize = cacheDAO.listLpush(keyBytes);
// 写入记录后实时返回当前List元素数
if(currentSize >= RedisCacheDAO.MAX_CACHE_SIZE) {
System.out.println("cache is full. " + Thread.currentThread().getName() + " ready to park.");
lock.lock();
condition.await();
System.out.println("cache is consuming. " + Thread.currentThread().getName() + " unparked.");
}
} catch (InterruptedException e) {
System.out.println(Thread.currentThread().getName() + " is interuupted.");
} finally {
if(lock.isLocked()) {
lock.unlock();
}
}
}
}
private String genKeyPair() {
// TODO 秘钥对桩
return "";
}
}
class KeyPairGeneratorThreadFactory implements ThreadFactory {
private final String threadGroupName;
private final AtomicInteger idSeq;
public KeyPairGeneratorThreadFactory(String threadGroupName, int maxSeq) {
this.threadGroupName = threadGroupName;
this.idSeq = new AtomicInteger(maxSeq);
}
@Override
public Thread newThread(Runnable r) {
int threadId = idSeq.getAndDecrement();
if(threadId < 0) {
throw new UnsupportedOperationException("thread number cannot be out of range");
}
return new Thread(r, threadGroupName + "_" + threadId);
}
}
}
复制代码Secret key pair generation monitor
/**
* 秘钥对生成定时调度
*/
public enum KeyPairsMonitor {
INSTANCE;
private final ReentrantLock reentrantLock;
private final List<Condition> conditionList;
private final RedisCacheDAO redisCacheDAO;
private final int coreSize;
KeyPairsMonitor() {
this.redisCacheDAO = new RedisCacheDAO();
this.reentrantLock = new ReentrantLock();
coreSize = Runtime.getRuntime().availableProcessors();
this.conditionList = new ArrayList<>(coreSize);
for( int i=0; i< coreSize; i++ ) {
conditionList.add(reentrantLock.newCondition());
}
}
/**
* 启动密钥生成任务,开启调度
*/
public void monitor() {
KeyPairExecutors executors = new KeyPairExecutors(reentrantLock, conditionList);
executors.start();
buildMonitorSchedule();
}
/**
* 构造定时任务
*/
private void buildMonitorSchedule() {
ScheduledExecutorService service = Executors.newSingleThreadScheduledExecutor();
service.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
int currentSize = redisCacheDAO.listLlen();
System.out.println("current cache size is: " + currentSize);
int executNum = 0;
if(currentSize <= RedisCacheDAO.HALF_MAX_CACHE_SIZE) {
System.out.println("current cache level is under 50% to ." + currentSize);
executNum = coreSize;
} else if(currentSize <= RedisCacheDAO.PERCENT_75_MAX_CACHE_SIZE) {
System.out.println("current cache level is under 75% to ." + currentSize);
executNum = coreSize >> 1;
}
for(int i=0; i < executNum; i++) {
try {
reentrantLock.lock();
conditionList.get(i).signal();
} catch (IllegalMonitorStateException e) {
// do nothing, condition no await
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
if(reentrantLock.isLocked()) {
reentrantLock.unlock();
}
}
}
}
}, 0, 5, TimeUnit.SECONDS);
}
复制代码List redis piling operation Operation:
public class RedisCacheDAO {
public static final String DEFAULT_KEYPAIE_CACHE_KEY = "keypaie_redis_list_rsa_byte";
public static final int MAX_CACHE_SIZE = 1 << 4;
public static final int HALF_MAX_CACHE_SIZE = MAX_CACHE_SIZE >> 1;
public static final int PERCENT_75_MAX_CACHE_SIZE = MAX_CACHE_SIZE - (MAX_CACHE_SIZE >> 2);
private String key;
private static final AtomicInteger count = new AtomicInteger(1);
public RedisCacheDAO() {
this.key = DEFAULT_KEYPAIE_CACHE_KEY;
}
public RedisCacheDAO(String key) {
this.key = key;
}
public int listLpush(String value) {
System.out.println(Thread.currentThread().getName() + " push value");
return count.addAndGet(1);
}
public int listLlen() {
return count.get();
}
public void listPop(int newValue) {
count.getAndSet(newValue);
}
}
复制代码Main method:
public static void main(String[] args) throws InterruptedException {
KeyPairsMonitor monitor = KeyPairsMonitor.INSTANCE;
monitor.monitor();
while (true) {
RedisCacheDAO dao = new RedisCacheDAO();
Thread.sleep(10);
dao.listPop(new Random().nextInt(RedisCacheDAO.MAX_CACHE_SIZE));
}
}
复制代码Think
According to the above modification program, at least to face traffic spikes can quickly buy high-core configuration ECS, rapid expansion. However, the dispatcher is dispersed run independently in each server, if the scheduling thread quits unexpectedly, the worker thread that server will never work.
Currently considered:
1 idea: a distributed task scheduling, adopt a unified dispatch center to schedule maintenance workers, but the complexity of the system will be greater than the current implementation;
Ideas 2: Use the timeout mechanism condition of autonomy, that is, set the timeout condition, and after a timeout automatically generated written, with the cache allkeys-random phase-out strategies to mitigate the risk of problems caused by the above.