GCC learning (5) dynamic library interface visibility

Why is C++ visibility support so important?

Simply put, it hides most of the previously public (unnecessary) ELF symbols, which means:

It greatly reduces the time for loading dynamic libraries ( DSO, Dynamic Shared Object ). After testing, the loading time of a large template library has changed from over 6 minutes to 6 seconds!
It enables the optimizer to produce better code. The indirect value of PLT (when the function call or variable access must be looked up through the global offset table (such as PIC code)) can be completely avoided, thus avoiding the pipeline pause on modern processors to a large extent, thus greatly speeding up The speed of the code. In addition, when most symbols are bound locally, they can be completely safely omitted (removed) through the entire DSO. This gives the inline a greater degree of freedom, especially the inline does not need to maintain a "just in case" entry point;
It reduces the size of the dynamic library by 5-20%. The format of the symbol table exported by ELF is quite space-consuming and can provide a complete error symbol name. If a large number of templates are used, it takes about 1000 bytes on average. C++ templates take up a lot of symbols, a typical C++ library can easily exceed 30,000 symbols, about 5-6Mb! Therefore, if you delete 60-80% of unnecessary symbols, DSO can be fractional megabytes!
Less chance of symbol conflict. With the support of this patch, the old trouble of two libraries that internally used the same symbol for different processing has finally passed. Hallelujah!

Although the library referenced above is an extreme case, the new visibility support reduces the exported symbol table from> 200,000 symbols to less than 18,000. The binary file size has also been reduced by 21Mb! Someone might suggest that the GNU linker version script can also do it. Maybe for C programs, this is correct, but for C++, it is not correct-unless you laboriously specify each symbol to make it public (and its complex name confusion), you must use wildcards , Because wildcards can cause a lot of false symbols. If you decide to change the name of the class or function, you must update the linker script. For the above library, a symbol table with less than 40,000 symbols cannot be obtained using the version script. In addition, using the linker version of the script does not allow GCC to optimize the code better.

Windows compatible

The non-Windows version of GCC cannot provide an interface similar to that __declspec（dllexport）used to mark the C/C++interface as a shared library, which frustrates those who deal with large portable applications on Windows and POSIX [2]. A good dynamic library interface design method. Writing good code is as important as designing the visibility of the class.

Although the syntax of Windows DLL and ELF DSO is different, notice that all codes under Windows choose whether to use dllimportor use when choosing macro compilation dllexport. We can re-use the DLL compilation support under Windows by simply patching the program. In fact, it only costs You complete this patch in 5 minutes.

Windows with different functional semantics here GCC reflected in: __declspec（dllexport）void（* foo）（void）and void（__declspec（dllexport）* foo）（void）the meaning expressed completely different, suggesting he will not be the property of the application of non-type warning in GCC.

How to use the new C++ visible attribute support?

In your header file, whenever you want your interface or API to become public, you only need to __attribute__ ((visibility ("default")))put it in front of the structure, class, and function declarations (it will be easier if you define macros), It does not need to be specified in the definition. Then, every time GCC compiles the source file, additional parameters are added -fvisibility=hiddento the make system. That's All! If you are throwing a shared boundary error, please refer to the "C++ exception problem" below to use the nm -C -Doutput to handle the difference before and after DSO.

#if defined _WIN32 || defined __CYGWIN__
  #ifdef BUILDING_DLL
    #ifdef __GNUC__
      #define DLL_PUBLIC __attribute__ ((dllexport))
    #else
      #define DLL_PUBLIC __declspec(dllexport) // Note: actually gcc seems to also supports this syntax.
    #endif
  #else
    #ifdef __GNUC__
      #define DLL_PUBLIC __attribute__ ((dllimport))
    #else
      #define DLL_PUBLIC __declspec(dllimport) // Note: actually gcc seems to also supports this syntax.
    #endif
  #endif
  #define DLL_LOCAL
#else
  #if __GNUC__ >= 4
    #define DLL_PUBLIC __attribute__ ((visibility ("default")))
    #define DLL_LOCAL  __attribute__ ((visibility ("hidden")))
  #else
    #define DLL_PUBLIC
    #define DLL_LOCAL
  #endif
#endif

extern "C" DLL_PUBLIC void function(int a);
class DLL_PUBLIC SomeClass
{
    
    
   int c;
   DLL_LOCAL void privateMethod();  // Only for use within this DSO
public:
   Person(int _c) : c(_c) {
    
     }
   static void foo(int a);
};

This is helpful for generating more optimized code: when you declare the definition outside the compilation unit, GCC cannot determine whether the current compilation unit is inside or outside the DSO. The worst case will be considered and the global offset table (GOT , Global Offset Table) this mechanism makes the dynamic link library bear the code space and additional relocation overhead. In order to reduce this overhead, we need to inform GCC of the visibility of the classes, structures, functions, or variables defined by the current DSO by manually specifying the hidden visibility attribute hidden visibility(that is, the DLL_LOCAL in the above example) in its header file . This will Let GCC optimize its code.

In order to solve the trouble of (specifying visibility every time), GCC has added options -fvisibility. Using -fvisibility = hidden, you will tell GCC that every declaration that is not explicitly marked as a visibility attribute has hidden visibility. Just like the example above, even for classes that are marked as visible (exported from DSO), you may still want to mark, for example, private members are hidden, so when they are called (from within DSO) the best code will be generated, in order to help you To convert the old code to use the new system, GCC now also supports the #pragma GCC visibility command:

extern void foo(int);
#pragma GCC visibility push(hidden)
extern void someprivatefunct(int);
#pragma GCC visibility pop

#pragma GCC visibilityStronger than -fvisibility. It also affects external declarations. -fvisibilityOnly affects the definition, so existing code can be recompiled with minimal changes. For C, this ratio is C ++more correct. C ++Interfaces tend to use affected -fvisibilityclasses.

Finally, a new command option -fvisibility-inlines-hidden. This will result in hidden visibility of all inline class member functions, resulting in a significant reduction in the exported symbol table and binary size, although not -fvisibility = hiddenas large as used . However, it -fvisibility-inlines-hiddencan be used without changing the source file. Unless you need to overwrite the inline whose address identifier is important to the function itself or the function's local static data, you must overwrite it.

C++ exception problem (please read!)

Using binary to catch user-defined types of exceptions, rather than the binary that caused the exception, requires a typeinfosearch. Go back and read the last statement again. This is the reason when abnormalities start to malfunction mysteriously! Just like functions and variables, types raised between multiple shared objects are public interfaces and must have default visibility. The obvious first step is to always mark all types that can be thrown across shared object boundaries as default visibility. You must do this, because even (for example) the type of exception code is implemented in DLL A, when DLL Bthrowing instance of the type, DLL Cthe catchhandler will look for the DLL B typeinfo.
But that's not all-it's getting harder and harder. By default, the symbol visibility is "default", but if the linker encounters only one hidden definition (only one definition), the typeinfo symbol will be permanently hidden (remember the C ++standard ODR-one definition rule). This is true for all symbols, but it is more likely to affect you through typeinfos. The typeinfo symbol of the class without vtable is defined on demand in the object file of each class that uses EH, and the definition is weak, so these definitions are merged into one copy when linking.

The result of this is that if you forget the preprocessor only defined in an object file, or at any time will not be thrown type is declared as an explicit public, it -fvisibility = hiddenmarks it as hidden in the target file , Thereby overriding all other definitions with default visibility, and cause typeinfodisappearing in the output binary file (then, any throw of that type will cause terminate() to be called in the captured binary file). Your binaries will be perfectly linked, and even if they don't work properly, they can still work.

Although it is good to warn about this, there are many reasonable reasons to keep the disposable types out of public view. Until the entire program optimization is added to GCC, the compiler does not know which exceptions to catch locally.

Other vaguely linked entities (such as static data members of class templates) may also have the same problem. If the class has hidden visibility, the data member can be instantiated in multiple DSOs and referenced separately, causing damage.

This problem also occurs when using classes as the operands of dynamic_cast. Make sure to export all such.

Hands-on teaching

The following instructions are how to add full support to your library to produce the highest quality code and minimize binary file size, load time, and link time. All new code should have this support from the beginning! And it's worth your time, especially in libraries that have strict requirements on speed, to fully implement it-this is a one-time investment of time and nothing more. However, although this is not recommended, you can add basic support to your library in a very short time.

In the main header file (or specific headers that will be included everywhere), put the following into the code. The code is taken from the aforementioned TnFOX library:

// Generic helper definitions for shared library support
#if defined _WIN32 || defined __CYGWIN__
  #define FOX_HELPER_DLL_IMPORT __declspec(dllimport)
  #define FOX_HELPER_DLL_EXPORT __declspec(dllexport)
  #define FOX_HELPER_DLL_LOCAL
#else
  #if __GNUC__ >= 4
    #define FOX_HELPER_DLL_IMPORT __attribute__ ((visibility ("default")))
    #define FOX_HELPER_DLL_EXPORT __attribute__ ((visibility ("default")))
    #define FOX_HELPER_DLL_LOCAL  __attribute__ ((visibility ("hidden")))
  #else
    #define FOX_HELPER_DLL_IMPORT
    #define FOX_HELPER_DLL_EXPORT
    #define FOX_HELPER_DLL_LOCAL
  #endif
#endif

// Now we use the generic helper definitions above to define FOX_API and FOX_LOCAL.
// FOX_API is used for the public API symbols. It either DLL imports or DLL exports (or does nothing for static build)
// FOX_LOCAL is used for non-api symbols.

#ifdef FOX_DLL // defined if FOX is compiled as a DLL
  #ifdef FOX_DLL_EXPORTS // defined if we are building the FOX DLL (instead of using it)
    #define FOX_API FOX_HELPER_DLL_EXPORT
  #else
    #define FOX_API FOX_HELPER_DLL_IMPORT
  #endif // FOX_DLL_EXPORTS
  #define FOX_LOCAL FOX_HELPER_DLL_LOCAL
#else // FOX_DLL is not defined: this means FOX is a static lib.
  #define FX_API
  #define FOX_LOCAL
#endif // FOX_DLL

Obviously, you may want to replace FOX with a prefix suitable for your library. For projects that also support Win32, you will find many of the above familiar things (you can reuse most of the Win32 macro mechanisms to support GCC). Explanation:

If defined _WIN32(automatically defined when compiling Windows)
If defined FOX_DLL_EXPORTS, we will compile our library and need the exported symbols. So you define in the compilation system FOX_DLL_EXPORTSto compile the FOX DLL library. MSVC defines the _EXPORTSending content in all IDEs (dito CMake default settings, please refer to CMake Wiki BuildingWinDLL).
If it is not defined FOX_DLL_EXPORTS(that is, the client uses the library), we will use the input library and symbols
If it is WIN32not defined (that is, compile GCC under Unix)
If __GNUC__>=4true, it means that the GCC version is greater than 4.0, so these new features are supported
For the non-template and non-static function definitions (header files and source files) in each library, determine whether it is public or internal
If the object is used in public form, use it FOX_APIto mark it like thisextern FOX_API PublicFunc()
If it is only used internally, use FOX_LOCALto mark it, like this extern FOX_LOCAL PublicFunc(). Remember that static functions do not need to be divided, nor do they need to be templated.
For each non-template class (header file and source file) defined in your library, decide whether it is public or internal
If it’s public, FOX_APImark it like thisFOX_API PublicClass
If used internally, FOX_LOCALmark it like thisFOX_LOCAL PublicClass
The member functions of the derived class that are not part of the interface, especially the member functions that are private and are not used by friends, should be marked with FOX_LOCAL respectively
In your build system (Makefile, etc.), you may want to add the -fvisibility=hidden and -fvisibility-inlines-hidden options to the command line arguments of each GCC invocation. Remember to thoroughly test your library afterwards, including all exceptions traversing the shared object boundary correctly.

If you want to see the difference before and after, please use the command nm -C -Dto list all the exported symbols in a non-hybrid form.

Extended reading: https://developer.ibm.com/technologies/systems/articles/au-aix-symbol-visibility/
[1] https://gcc.gnu.org/wiki/Visibility
[2] "__declspec" is A special keyword in Microsoft C++, it can extend standard C++ with some attributes. These attributes are: align, allocate, deprecated, dllexport, dllimport, naked, noinline, noreturn, nothrow, novtable, selectany, thread, property and uuid