Windows7 x64 system call analysis

table of Contents

• 0x01 Syscall & Sysret
  •  32 with distinction sysenter & int 2e   
• 0x02 KiSystemCall64
  •  Construction TrapFrame
  •  _kthread->header->DebugActive.ActiveDR7|Instrumented
  •  user-mode scheduling (??? This part is not yet clear)
  •  SSDT & ShadowSSDT API address calculation
  •  KiInitiateUserApc Apc min 发
  •  InstrumentationCallback user-level callbacks

 

---

0x01 Syscall & Sysret

• Syscall

 

Syscall

IF (CS.L ≠ 1 ) or (IA32_EFER.LMA ≠ 1) or (IA32_EFER.SCE ≠ 1)
(* Not in 64-Bit Mode or SYSCALL/SYSRET not enabled in IA32_EFER *)
THEN #UD;
FI;
RCX ← RIP; (* Will contain address of next instruction *)
RIP ← IA32_LSTAR;
R11 ← RFLAGS;
RFLAGS ← RFLAGS AND NOT(IA32_FMASK);
CS.Selector ← IA32_STAR[47:32] AND FFFCH (* Operating system provides CS; RPL forced to 0 *)
(* Set rest of CS to a fixed value *)
CS.Base ← 0; (* Flat segment *)
CS.Limit ← FFFFFH; (* With 4-KByte granularity, implies a 4-GByte limit *)
CS.Type ← 11; (* Execute/read code, accessed *)
CS.S ← 1;
CS.DPL ← 0;
CS.P ← 1;
CS.L ← 1; (* Entry is to 64-bit mode *)
CS.D ← 0; (* Required if CS.L = 1 *)
CS.G ← 1; (* 4-KByte granularity *)
CPL ← 0;

　　It can be seen from the pseudocode above Syscall, rcx stored is the address of the next instruction, the windows will rcx before Syscall stored in r10. r11 is stored rflags. rip is obtained from the IA32_LSTAR, cs selectors from IA32_STAR [ : 47 32] is read, but the other segments cs attribute will not read in the memory according to the selected sub, a fixed value instead of the set, thus OS obligation to choose the corresponding segment descriptor segments and corresponding fixed value set. The above pseudo-code can be found, and no stack switch, it is necessary to conduct their own OS system call stack switching routine.

 

• Sysret

IF (CS.L ≠ 1 ) or (IA32_EFER.LMA ≠ 1) or (IA32_EFER.SCE ≠ 1)
(* Not in 64-Bit Mode or SYSCALL/SYSRET not enabled in IA32_EFER *)
THEN #UD; FI;
IF (CPL ≠ 0) THEN #GP(0); FI;
IF (operand size is 64-bit)
THEN (* Return to 64-Bit Mode *)
IF (RCX is not canonical) THEN #GP(0);
RIP ← RCX;
ELSE (* Return to Compatibility Mode *)
RIP ← ECX;
FI;
RFLAGS ← (R11 & 3C7FD7H) | 2; (* Clear RF, VM, reserved bits; set bit 2 *)
IF (operand size is 64-bit)
THEN CS.Selector ← IA32_STAR[63:48]+16;
ELSE CS.Selector ← IA32_STAR[63:48];
FI;
CS.Selector ← CS.Selector OR 3; (* RPL forced to 3 *)
(* Set rest of CS to a fixed value *)
CS.Base ← 0; (* Flat segment *)
CS.Limit ← FFFFFH; (* With 4-KByte granularity, implies a 4-GByte limit *)
CS.Type ← 11; (* Execute/read code, accessed *)
CS.S ← 1;
CS.DPL ← 3;
CS.P ← 1;
IF (operand size is 64-bit)
THEN (* Return to 64-Bit Mode *)
CS.L ← 1; (* 64-bit code segment *)
CS.D ← 0; (* Required if CS.L = 1 *)
ELSE (* Return to Compatibility Mode *)
CS.L ← 0; (* Compatibility mode *)
CS.D ← 1; (* 32-bit code segment *)
FI;
CS.G ← 1; (* 4-KByte granularity *)
CPL ← 3;
SS.Selector ← (IA32_STAR[63:48]+8) OR 3; (* RPL forced to 3 *)
(* Set rest of SS to a fixed value *)
SS.Base ← 0; (* Flat segment *)
SS.Limit ← FFFFFH; (* With 4-KByte granularity, implies a 4-GByte limit *)
SS.Type ← 3; (* Read/write data, accessed *)
SS.S ← 1;
SS.DPL ← 3;
SS.P ← 1;
SS.B ← 1; (* 32-bit stack segment*)
SS.G ← 1; (* 4-KByte granularity *)

　　SYSRET loads the CS and SS selectors with values derived from bits 63:48 of the IA32_STAR MSR. However, the CS and SS descriptor caches are not loaded from the descriptors (in GDT or LDT) referenced by those selectors. Instead, the descriptor caches are loaded with fixed values.

• rip<-rcx
• rflags<-(R11 & 3C7FD7H) | 2