MIF_E31221222/sigap-mobile/docs/face_detection_troubleshooting.mdx

# Face Detection Troubleshooting Guide

## Common Detection Issue

If you're experiencing issues with face detection failing or consistently returning zero faces (`Detected 0 faces`), this guide will help you understand and resolve the problem.

## Understanding the Problem

The ML Kit face detection can fail for several reasons, but one of the most common is related to image orientation and processing. This is especially problematic on iOS devices where camera orientation metadata isn't correctly passed to the ML Kit detector.

### Common Error Logs

```
[LIVENESS_CONTROLLER] Detected 0 faces
[LIVENESS_CONTROLLER] Detected 0 faces
[LIVENESS_CONTROLLER] Detected 0 faces
```

When you see these logs repeatedly, it means that the face detector is unable to identify any faces in the camera frames, even when a face is clearly visible to the user.

## Root Causes

1. **Image Orientation Issues**: On iOS especially, the orientation metadata in images captured from the camera may not be correctly interpreted by ML Kit.

2. **Format Compatibility**: Different platforms use different image formats (YUV420, NV21, BGRA8888), and improper handling can lead to detection failures.

3. **Image Processing**: Camera frames need proper conversion before they can be processed by ML Kit.

4. **Resolution Issues**: Low-resolution images can make face detection difficult.

## Solution Implementation

Our solution implements several fixes based on common patterns found on GitHub and in the ML Kit community:

### 1. Platform-Specific Image Format Handling

```dart
cameraController = CameraController(
  frontCamera,
  ResolutionPreset.high, // Higher resolution
  enableAudio: false,
  imageFormatGroup: Platform.isIOS ? ImageFormatGroup.bgra8888 : ImageFormatGroup.yuv420,
);
```

We use different image format groups for iOS and Android to ensure compatibility.

### 2. Proper Image Rotation

For real-time detection, we carefully calculate the correct rotation:

```dart
if (Platform.isIOS) {
  // iOS-specific rotation handling
  rotation = InputImageRotationValue.fromRawValue(sensorOrientation);
} else if (Platform.isAndroid) {
  // Android-specific rotation calculation
  var rotationCompensation = orientations[cameraController!.value.deviceOrientation];
  if (rotationCompensation == null) return null;

  if (camera.lensDirection == CameraLensDirection.front) {
    rotationCompensation = (sensorOrientation + rotationCompensation) % 360;
  } else {
    rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360;
  }

  rotation = InputImageRotationValue.fromRawValue(rotationCompensation);
}
```

### 3. Platform-Specific Image Processing

For Android, we handle YUV420 format which requires proper handling of multiple image planes:

```dart
if (Platform.isAndroid) {
  // Android requires proper YUV420 handling with all planes
  final plane1 = image.planes[0];
  final plane2 = image.planes[1];
  final plane3 = image.planes[2];

  return InputImage.fromBytes(
    bytes: Uint8List.fromList([...plane1.bytes, ...plane2.bytes, ...plane3.bytes]),
    metadata: InputImageMetadata(
      size: Size(image.width.toDouble(), image.height.toDouble()),
      rotation: rotation,
      format: format,
      bytesPerRow: plane1.bytesPerRow,
    ),
  );
}
```

### 4. Fix for Captured Images on iOS

When capturing a still image (as opposed to real-time stream processing), we apply a special fix for iOS to ensure proper orientation:

```dart
Future<XFile> _processAndFixImageOrientation(XFile originalImage) async {
  if (Platform.isIOS) {
    try {
      // Get temp directory
      final directory = await getApplicationDocumentsDirectory();
      final path = directory.path;
      final filename = 'processed_${DateTime.now().millisecondsSinceEpoch}.jpg';
      final outputPath = '$path/$filename';

      // Read and decode
      final imageBytes = await originalImage.readAsBytes();
      final originalDecodedImage = imglib.decodeImage(imageBytes);

      // Fix orientation
      final orientedImage = imglib.bakeOrientation(originalDecodedImage);

      // Save
      final processedImageFile = File(outputPath);
      await processedImageFile.writeAsBytes(imglib.encodeJpg(orientedImage));

      return XFile(processedImageFile.path);
    } catch (e) {
      return originalImage; // Fall back if processing fails
    }
  }

  // No fix needed for Android usually
  return originalImage;
}
```

### 5. Image Verification

We added a verification step that checks if a captured image actually contains a face:

```dart
Future<bool> _verifyFaceInImage(XFile image) async {
  try {
    final inputImage = InputImage.fromFilePath(image.path);
    final faces = await faceDetector.processImage(inputImage);
    return faces.isNotEmpty;
  } catch (e) {
    return false;
  }
}
```

## How These Fixes Work

1. **For iOS**: We explicitly handle orientation using the `bakeOrientation` method from the `image` package, which ensures that the image pixels are correctly rotated in memory rather than relying on metadata.

2. **For Android**: We handle all three YUV planes and ensure the correct rotation is calculated based on device orientation and camera direction.

3. **For Both Platforms**: We use higher resolution settings and proper image format groups.

## Additional Dependencies Required

To implement these fixes, make sure to add these dependencies to your `pubspec.yaml`:

```yaml
dependencies:
  image: ^4.0.17 # For orientation baking
  path_provider: ^2.0.15 # For accessing temporary directories
```

## Common Pitfalls

1. **Missing Planes**: Always check that image planes are available before processing.
2. **Rotation Calculation**: Be careful with rotation calculation, as different platforms and camera directions require different formulas.
3. **Memory Usage**: Higher resolution images improve detection but consume more memory. Monitor performance.
4. **Permissions**: Ensure camera permissions are properly granted.

## References

- [Google ML Kit Documentation](https://developers.google.com/ml-kit)
- [Flutter Camera Plugin](https://pub.dev/packages/camera)
- [Image Package Documentation](https://pub.dev/packages/image)
- [Face Detection GitHub Issue - Flutter](https://github.com/flutter/flutter/issues/79116)

## Conclusion

Face detection issues can often be fixed by properly handling image orientation and format. The implementation details vary by platform, so it's important to have platform-specific handling. By applying these fixes, you should see a significant improvement in face detection reliability.