The camera's architecture is consistent with the overall architecture of the Android system, as shown in the figure below. This article mainly describes it from the following four aspects.
- Framework：Camera.java
- Android Runtime：android_hardware_Camera.cpp
- Library：Camera Client和Camera Service
- HAL：CameraHardwareInterface
Camera system architecture
一、Framework：Camera.java
Camera is a class directly used by the application layer software, covering all interfaces for operating the camera such as startup, preview, shooting and closing. The path of Camera.java in the Android source code is: framework/base/core/java/android/hardware. In order to illustrate the architecture of the entire Camera system, we will not analyze the functions of Camera.java horizontally here. Let's start with the open() method:
public static Camera open() {
int numberOfCameras = getNumberOfCameras();
CameraInfo cameraInfo = new CameraInfo();
for (int i = 0; i < numberOfCameras; i++) {
getCameraInfo(i, cameraInfo);
if (cameraInfo.facing == CameraInfo.CAMERA_FACING_BACK) {
return new Camera(i);
}
}
return null;
}
The open() method needs to pay attention to the following points:
- getNumberOfCameras is a native method, implemented in android_hardware_Camera.cpp;
- CameraInfo is a static inner class defined by Camera, including facing, orientation, canDisableShutterSound;
- getCameraInfo internally calls the native method _getCameraInfo to obtain camera information;
- open() starts the rear camera by default (CAMERA_FACING_BACK).
/** used by Camera#open, Camera#open(int) */
Camera(int cameraId) {
int err = cameraInitNormal(cameraId);
if (checkInitErrors(err)) {
switch(err) {
case EACCESS:
throw new RuntimeException("Fail to connect to camera service");
case ENODEV:
throw new RuntimeException("Camera initialization failed");
default:
// Should never hit this.
throw new RuntimeException("Unknown camera error");
}
}
}
The core implementation of the Camera constructor is in cameraInitNormal, cameraInitNormal calls cameraInitVersion, and passes in the parameters cameraId and CAMERA_HAL_API_VERSION_NORMAL_CONNECT, which represents the version of HAL.
private int cameraInitVersion(int cameraId, int halVersion) {
……
String packageName = ActivityThread.currentPackageName();
return native_setup(new WeakReference<Camera>(this), cameraId, halVersion, packageName);
}
cameraInitNormal calls the native method native_setup(), which enters android_hardware_Camera.cpp. The signature of native_setup() is as follows:
private native final int native_setup(Object camera_this, int cameraId, int halVersion, String packageName);
二、Android Runtime：android_hardware_Camera.cpp
native_setup() is dynamically registered to JNI, and the android_hardware_Camera_native_setup() method is called through JNI.
static JNINativeMethod camMethods[] = {
……
{ "native_setup", "(Ljava/lang/Object;ILjava/lang/String;)V",
(void*)android_hardware_Camera_native_setup }
……
};
The focus of JNI is the implementation of the android_hardware_Camera_native_setup() method:
// connect to camera service
static jint android_hardware_Camera_native_setup(JNIEnv *env, jobject thiz,
jobject weak_this, jint cameraId, jint halVersion, jstring clientPackageName)
{
// Convert jstring to String16
const char16_t *rawClientName = env->GetStringChars(clientPackageName, NULL);
jsize rawClientNameLen = env->GetStringLength(clientPackageName);
String16 clientName(rawClientName, rawClientNameLen);
env->ReleaseStringChars(clientPackageName, rawClientName);
sp<Camera> camera;
if (halVersion == CAMERA_HAL_API_VERSION_NORMAL_CONNECT) {
// Default path: hal version is don't care, do normal camera connect.
camera = Camera::connect(cameraId, clientName,
Camera::USE_CALLING_UID);
} else {
jint status = Camera::connectLegacy(cameraId, halVersion, clientName,
Camera::USE_CALLING_UID, camera);
if (status != NO_ERROR) {
return status;
}
}
if (camera == NULL) {
return -EACCES;
}
// make sure camera hardware is alive
if (camera->getStatus() != NO_ERROR) {
return NO_INIT;
}
jclass clazz = env->GetObjectClass(thiz);
if (clazz == NULL) {
// This should never happen
jniThrowRuntimeException(env, "Can't find android/hardware/Camera");
return INVALID_OPERATION;
}
// We use a weak reference so the Camera object can be garbage collected.
// The reference is only used as a proxy for callbacks.
sp<JNICameraContext> context = new JNICameraContext(env, weak_this, clazz, camera);
context->incStrong((void*)android_hardware_Camera_native_setup);
camera->setListener(context);
// save context in opaque field
env->SetLongField(thiz, fields.context, (jlong)context.get());
return NO_ERROR;
}
The android_hardware_Camera_native_setup() method requests to connect to the CameraService service by calling the Camera::connect() method. In the reference:
- clientName is obtained by converting clientPackageName from jstring to String16 format;
- Camera::USE_CALLING_UID is an enumeration type defined in Camera.h, and its value is ICameraService::USE_CALLING_UID (also an enumeration type, with a value of -1).
Camera::connect() is located in Camera.cpp, thus entering the Library layer.
3. Library: Camera Client and Camera Service
As shown in the above architecture diagram, the three classes ICameraService.h, ICameraClient.h and ICamera.h define the interface and architecture of the Camera. The two files ICameraService.cpp and Camera.cpp are used for the implementation of the Camera architecture. The function is implemented by calling the hardware-related interface at the lower layer. Camera.h is the interface of the Camera system to the upper layer.
Specifically, the Camera class inherits the template class CameraBase, and Camera::connect() calls the connect() method in CameraBase.cpp.
sp<Camera> Camera::connect(int cameraId, const String16& clientPackageName,
int clientUid)
{
return CameraBaseT::connect(cameraId, clientPackageName, clientUid);
}
CameraBase actually inherits the DeathRecipient inner class of IBinder, and DeathRecipient virtual inherits from RefBase. RefBase is the reference counting basic class in Android, which defines incStrong, decStrong, incWeak, and decWeak and other pointer manipulation functions involving sp/wp. Of course, this is far away.
template <typename TCam>
struct CameraTraits {
};
template <typename TCam, typename TCamTraits = CameraTraits<TCam> >
class CameraBase : public IBinder::DeathRecipient
{
public:
static sp<TCam> connect(int cameraId,
const String16& clientPackageName,
int clientUid);
……
}
class DeathRecipient : public virtual RefBase
{
public:
virtual void binderDied(const wp<IBinder>& who) = 0;
};
Back to the implementation of Camera::connect(), where new TCam(cameraId) generates a BnCameraClient object, and BnCameraClient is defined in the ICameraClient.h file and inherits from the template class BnInterface. The getCameraService() method returns the service proxy BpCameraService of CameraService, which also inherits from the template class BnInterface. Then send the CONNECT command through Binder communication. When BnCameraService receives the CONNECT command, call the connect() member function of CameraService to do the corresponding processing.
template <typename TCam, typename TCamTraits>
sp<TCam> CameraBase<TCam, TCamTraits>::connect(int cameraId,
const String16& clientPackageName,
int clientUid)
{
ALOGV("%s: connect", __FUNCTION__);
sp<TCam> c = new TCam(cameraId); // BnCameraClient
sp<TCamCallbacks> cl = c;
status_t status = NO_ERROR;
const sp<ICameraService>& cs = getCameraService(); // BpCameraService
if (cs != 0) {
TCamConnectService fnConnectService = TCamTraits::fnConnectService;
status = (cs.get()->*fnConnectService)(cl, cameraId, clientPackageName, clientUid,
/*out*/ c->mCamera);
}
if (status == OK && c->mCamera != 0) {
c->mCamera->asBinder()->linkToDeath(c);
c->mStatus = NO_ERROR;
} else {
ALOGW("An error occurred while connecting to camera: %d", cameraId);
c.clear();
}
return c;
}
class BnCameraClient: public BnInterface<ICameraClient>
{
public:
virtual status_t onTransact( uint32_t code,
const Parcel& data,
Parcel* reply,
uint32_t flags = 0);
};
class BpCameraService: public BpInterface<ICameraService>
{
public:
BpCameraService(const sp<IBinder>& impl)
: BpInterface<ICameraService>(impl)
{
}
……
}
Note: The connect() function is declared as a pure virtual function in the parent class ICameraService of BpCameraService and BnCameraService, and its implementation is given in BpCameraService and CameraService respectively. As a proxy class, BpCameraService provides an interface to the client and is actually implemented in a subclass of BnCameraService in CameraService.
In BpCameraService, the connect() function is implemented as follows:
// connect to camera service (android.hardware.Camera)
virtual status_t connect(const sp<ICameraClient>& cameraClient, int cameraId,
const String16 &clientPackageName, int clientUid,
/*out*/
sp<ICamera>& device)
{
Parcel data, reply;
data.writeInterfaceToken(ICameraService::getInterfaceDescriptor());
data.writeStrongBinder(cameraClient->asBinder());
data.writeInt32(cameraId);
data.writeString16(clientPackageName);
data.writeInt32(clientUid);
// BpBinder的transact()函数向IPCThreadState实例发送消息，通知其有消息要发送给binder driver
remote()->transact(BnCameraService::CONNECT, data, &reply);
if (readExceptionCode(reply)) return -EPROTO;
status_t status = reply.readInt32();
if (reply.readInt32() != 0) {
device = interface_cast<ICamera>(reply.readStrongBinder()); // client端读出server返回的bind
}
return status;
}
First, convert the passed Camera object cameraClient into IBinder type, write the parameters of the call to Parcel (which can be understood as a pipeline for Binder communication), send a message through the transact() function of BpBinder, and then BnCameraService responds to the connection, and finally Just wait for the server to return, and if successful, generate a BpCamera instance.
The real server-side response is implemented in the onTransact() function of BnCameraService, which is responsible for unpacking the received Parcel and executing the client-side request method.
status_t BnCameraService::onTransact(
uint32_t code, const Parcel& data, Parcel* reply, uint32_t flags)
{
switch(code) {
……
case CONNECT: {
CHECK_INTERFACE(ICameraService, data, reply);
sp<ICameraClient> cameraClient =
interface_cast<ICameraClient>(data.readStrongBinder()); // 使用Camera的Binder对象生成Camera客户代理BpCameraClient实例
int32_t cameraId = data.readInt32();
const String16 clientName = data.readString16();
int32_t clientUid = data.readInt32();
sp<ICamera> camera;
status_t status = connect(cameraClient, cameraId,
clientName, clientUid, /*out*/camera); // 将生成的BpCameraClient对象作为参数传递到CameraService的connect()函数中
reply->writeNoException();
reply->writeInt32(status); // 将BpCamera对象以IBinder的形式打包到Parcel中返回
if (camera != NULL) {
reply->writeInt32(1);
reply->writeStrongBinder(camera->asBinder());
} else {
reply->writeInt32(0);
}
return NO_ERROR;
} break;
……
}
}
The main processing includes:
- Generate an instance of Camera client proxy BpCameraClient through the Binder object of Camera in data;
- Pass the generated BpCameraClient object as a parameter to the connect() function of CameraService (/frameworks/av/services/camera/libcameraservice/CameraService.cpp), which will return a BpCamera instance;
- Pack the above instance objects into Parcel in the form of IBinder and return them.
Finally, the BpCamera instance is returned by the CameraService::connect() function. The core of CameraService::connect() is to call the connectHelperLocked() function to create different client instances according to different HAL API versions (there seems to be no connectHelperLocked() function in earlier versions, but the functions are basically similar).
status_t CameraService::connectHelperLocked(
/*out*/
sp<Client>& client,
/*in*/
const sp<ICameraClient>& cameraClient,
int cameraId,
const String16& clientPackageName,
int clientUid,
int callingPid,
int halVersion,
bool legacyMode) {
int facing = -1;
int deviceVersion = getDeviceVersion(cameraId, &facing);
if (halVersion < 0 || halVersion == deviceVersion) {
// Default path: HAL version is unspecified by caller, create CameraClient
// based on device version reported by the HAL.
switch(deviceVersion) {
case CAMERA_DEVICE_API_VERSION_1_0:
client = new CameraClient(this, cameraClient,
clientPackageName, cameraId,
facing, callingPid, clientUid, getpid(), legacyMode);
break;
case CAMERA_DEVICE_API_VERSION_2_0:
case CAMERA_DEVICE_API_VERSION_2_1:
case CAMERA_DEVICE_API_VERSION_3_0:
case CAMERA_DEVICE_API_VERSION_3_1:
case CAMERA_DEVICE_API_VERSION_3_2:
client = new Camera2Client(this, cameraClient,
clientPackageName, cameraId,
facing, callingPid, clientUid, getpid(), legacyMode);
break;
case -1:
ALOGE("Invalid camera id %d", cameraId);
return BAD_VALUE;
default:
ALOGE("Unknown camera device HAL version: %d", deviceVersion);
return INVALID_OPERATION;
}
} else {
// A particular HAL version is requested by caller. Create CameraClient
// based on the requested HAL version.
if (deviceVersion > CAMERA_DEVICE_API_VERSION_1_0 &&
halVersion == CAMERA_DEVICE_API_VERSION_1_0) {
// Only support higher HAL version device opened as HAL1.0 device.
client = new CameraClient(this, cameraClient,
clientPackageName, cameraId,
facing, callingPid, clientUid, getpid(), legacyMode);
} else {
// Other combinations (e.g. HAL3.x open as HAL2.x) are not supported yet.
ALOGE("Invalid camera HAL version %x: HAL %x device can only be"
" opened as HAL %x device", halVersion, deviceVersion,
CAMERA_DEVICE_API_VERSION_1_0);
return INVALID_OPERATION;
}
}
status_t status = connectFinishUnsafe(client, client->getRemote());
if (status != OK) {
// this is probably not recoverable.. maybe the client can try again
return status;
}
mClient[cameraId] = client;
LOG1("CameraService::connect X (id %d, this pid is %d)", cameraId,
getpid());
return OK;
}
It can be seen that CameraClient is used for instantiation before CAMERA_DEVICE_API_VERSION_2_0, and Camera2Client is used for instantiation after that. Taking CameraClient as an example, its initialize() function is as follows:
status_t CameraClient::initialize(camera_module_t *module) {
int callingPid = getCallingPid();
status_t res;
LOG1("CameraClient::initialize E (pid %d, id %d)", callingPid, mCameraId);
// Verify ops permissions
res = startCameraOps();
if (res != OK) {
return res;
}
char camera_device_name[10];
snprintf(camera_device_name, sizeof(camera_device_name), "%d", mCameraId);
mHardware = new CameraHardwareInterface(camera_device_name);
res = mHardware->initialize(&module->common);
if (res != OK) {
ALOGE("%s: Camera %d: unable to initialize device: %s (%d)",
__FUNCTION__, mCameraId, strerror(-res), res);
mHardware.clear();
return res;
}
mHardware->setCallbacks(notifyCallback,
dataCallback,
dataCallbackTimestamp,
(void *)(uintptr_t)mCameraId);
// Enable zoom, error, focus, and metadata messages by default
enableMsgType(CAMERA_MSG_ERROR | CAMERA_MSG_ZOOM | CAMERA_MSG_FOCUS |
CAMERA_MSG_PREVIEW_METADATA | CAMERA_MSG_FOCUS_MOVE);
LOG1("CameraClient::initialize X (pid %d, id %d)", callingPid, mCameraId);
return OK;
}
In the above functions, mainly pay attention to the following processes:
The bold code CameraHardwareInterface creates a new Camera hardware interface, of course, camera_device_name is the camera device name;
- mHardware->initialize(&module->common) calls the initialization method of the underlying hardware;
- mHardware->setCallbacks registers the callback function at CamerService with HAL.
- CameraHardwareInterface defines the hardware abstraction characteristics of the Camera, thus entering the HAL.
四、HAL：CameraHardwareInterface
The role of CameraHardwareInterface is to link Camera Server and V4L2. By implementing CameraHardwareInterface, the impact of different drivers on Camera Server can be shielded. CameraHardwareInterface also virtually inherits from RefBase.
class CameraHardwareInterface : public virtual RefBase {
public:
CameraHardwareInterface(const char *name)
{
mDevice = 0;
mName = name;
}
……
}
CameraHardwareInterface includes a control channel and a data channel. The control channel is used to process functions such as start/stop of preview and video acquisition, taking photos, and autofocus. The data channel obtains data such as preview, video recording, and autofocus through callback functions. When you need to support new hardware, you need to inherit from CameraHardwareInterface to achieve the corresponding function. The public methods provided by CameraHardwareInterface are as follows:
CameraHardwareInterface
In the previous section, the initialize() function called mHardware->initialize and mHardware->setCallbacks. Let's take a look at the implementation of CameraHardwareInterface.h.
status_t initialize(hw_module_t *module)
{
ALOGI("Opening camera %s", mName.string());
camera_module_t *cameraModule = reinterpret_cast<camera_module_t *>(module);
camera_info info;
status_t res = cameraModule->get_camera_info(atoi(mName.string()), &info);
if (res != OK) return res;
int rc = OK;
if (module->module_api_version >= CAMERA_MODULE_API_VERSION_2_3 &&
info.device_version > CAMERA_DEVICE_API_VERSION_1_0) {
// Open higher version camera device as HAL1.0 device.
rc = cameraModule->open_legacy(module, mName.string(),
CAMERA_DEVICE_API_VERSION_1_0,
(hw_device_t **)&mDevice);
} else {
rc = CameraService::filterOpenErrorCode(module->methods->open(
module, mName.string(), (hw_device_t **)&mDevice));
}
if (rc != OK) {
ALOGE("Could not open camera %s: %d", mName.string(), rc);
return rc;
}
initHalPreviewWindow();
return rc;
}
In the initialize() method, the camera module is opened through cameraModule->open_legacy, and initHalPreviewWindow() is used to initialize the preview related stream opspreview_stream_ops, and initialize the hal preview window.
void initHalPreviewWindow()
{
mHalPreviewWindow.nw.cancel_buffer = __cancel_buffer;
mHalPreviewWindow.nw.lock_buffer = __lock_buffer;
mHalPreviewWindow.nw.dequeue_buffer = __dequeue_buffer;
mHalPreviewWindow.nw.enqueue_buffer = __enqueue_buffer;
mHalPreviewWindow.nw.set_buffer_count = __set_buffer_count;
mHalPreviewWindow.nw.set_buffers_geometry = __set_buffers_geometry;
mHalPreviewWindow.nw.set_crop = __set_crop;
mHalPreviewWindow.nw.set_timestamp = __set_timestamp;
mHalPreviewWindow.nw.set_usage = __set_usage;
mHalPreviewWindow.nw.set_swap_interval = __set_swap_interval;
mHalPreviewWindow.nw.get_min_undequeued_buffer_count =
__get_min_undequeued_buffer_count;
}
/** Set the notification and data callbacks */
void setCallbacks(notify_callback notify_cb,
data_callback data_cb,
data_callback_timestamp data_cb_timestamp,
void* user)
{
mNotifyCb = notify_cb;
mDataCb = data_cb;
mDataCbTimestamp = data_cb_timestamp;
mCbUser = user;
ALOGV("%s(%s)", __FUNCTION__, mName.string());
if (mDevice->ops->set_callbacks) {
mDevice->ops->set_callbacks(mDevice,
__notify_cb,
__data_cb,
__data_cb_timestamp,
__get_memory,
this);
}
}
In set_callbacks, __notify_cb, __data_cb, __data_cb_timestamp, and __get_memory are message callbacks, data callbacks, timestamp callbacks, and callbacks for memory-related operations, respectively.
The above briefly analyzes the responses of the Framework, ART, Library and HAL layers after the application layer calls Camera.open() to illustrate the overall architecture of the Camera system in Android, hoping to be of some help to the readers.
Author: yhthu
Link: https://www.jianshu.com/p/592b08e2e00e
Source : Jianshu The
copyright belongs to the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.