如何在屏幕上绘制标签:MLKit对象检测

这是我之前提出的一个问题的延伸。

我正在使用最新的MLKit对象检测版本(不需要firebase)。我使用了一个自定义模型和CameraX来检测对象并对其进行标记/获取信息。

目前,根据我现有的代码,它可以检测到区域内有对象存在,但存在以下问题:

  1. 没有显示标签或边界框;
  2. 一次只能检测一个对象;
  3. 一旦检测到一个对象,应用就不会“改变”(即当我移动手机,试图检测另一个对象时,显示屏上没有任何变化)。

这是我的代码:

package com.example.mlkitobjecttest;import androidx.annotation.NonNull;import androidx.appcompat.app.AppCompatActivity;import androidx.camera.core.Camera;import androidx.camera.core.CameraSelector;import androidx.camera.core.CameraX;import androidx.camera.core.ImageAnalysis;import androidx.camera.core.ImageProxy;import androidx.camera.core.Preview;import androidx.camera.core.impl.PreviewConfig;import androidx.camera.lifecycle.ProcessCameraProvider;import androidx.camera.view.PreviewView;import androidx.core.app.ActivityCompat;import androidx.core.content.ContextCompat;import androidx.lifecycle.LifecycleOwner;import android.content.pm.PackageManager;import android.graphics.Rect;import android.media.Image;import android.os.Bundle;import android.text.Layout;import android.util.Rational;import android.util.Size;import android.view.View;import android.widget.TextView;import android.widget.Toast;import com.google.android.gms.tasks.OnFailureListener;import com.google.android.gms.tasks.OnSuccessListener;import com.google.common.util.concurrent.ListenableFuture;import com.google.mlkit.common.model.LocalModel;import com.google.mlkit.vision.common.InputImage;import com.google.mlkit.vision.objects.DetectedObject;import com.google.mlkit.vision.objects.ObjectDetection;import com.google.mlkit.vision.objects.ObjectDetector;import com.google.mlkit.vision.objects.custom.CustomObjectDetectorOptions;import org.w3c.dom.Text;import java.util.List;import java.util.concurrent.ExecutionException;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;public class MainActivity extends AppCompatActivity {    private class YourAnalyzer implements ImageAnalysis.Analyzer {        @Override        @androidx.camera.core.ExperimentalGetImage        public void analyze(ImageProxy imageProxy) {            Image mediaImage = imageProxy.getImage();            if (mediaImage != null) {                InputImage image =                        InputImage.fromMediaImage(mediaImage, imageProxy.getImageInfo().getRotationDegrees());                // Pass image to an ML Kit Vision API                // ...                LocalModel localModel =                        new LocalModel.Builder()                                .setAssetFilePath("mobilenet_v1_1.0_128_quantized_1_default_1.tflite")                                // or .setAbsoluteFilePath(absolute file path to tflite model)                                .build();                CustomObjectDetectorOptions customObjectDetectorOptions =                        new CustomObjectDetectorOptions.Builder(localModel)                                .setDetectorMode(CustomObjectDetectorOptions.SINGLE_IMAGE_MODE)                                .enableMultipleObjects()                                .enableClassification()                                .setClassificationConfidenceThreshold(0.5f)                                .setMaxPerObjectLabelCount(3)                                .build();                ObjectDetector objectDetector =                        ObjectDetection.getClient(customObjectDetectorOptions);                objectDetector                        .process(image)                        .addOnFailureListener(new OnFailureListener() {                            @Override                            public void onFailure(@NonNull Exception e) {                                //Toast.makeText(getApplicationContext(), "Fail. Sad!", Toast.LENGTH_SHORT).show();                                //textView.setText("Fail. Sad!");                                imageProxy.close();                            }                        })                        .addOnSuccessListener(new OnSuccessListener<List<DetectedObject>>() {                            @Override                            public void onSuccess(List<DetectedObject> results) {                                for (DetectedObject detectedObject : results) {                                    Rect box = detectedObject.getBoundingBox();                                    for (DetectedObject.Label label : detectedObject.getLabels()) {                                        String text = label.getText();                                        int index = label.getIndex();                                        float confidence = label.getConfidence();                                        textView.setText(text);                                }}                                imageProxy.close();                            }                        });            }            //ImageAnalysis.Builder.fromConfig(new ImageAnalysisConfig).setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST);        }    }    PreviewView prevView;    private ListenableFuture<ProcessCameraProvider> cameraProviderFuture;    private ExecutorService executor = Executors.newSingleThreadExecutor();    TextView textView;    private int REQUEST_CODE_PERMISSIONS = 101;    private String[] REQUIRED_PERMISSIONS = new String[]{"android.permission.CAMERA"};   /* @NonNull    @Override    public CameraXConfig getCameraXConfig() {        return CameraXConfig.Builder.fromConfig(Camera2Config.defaultConfig())                .setCameraExecutor(ContextCompat.getMainExecutor(this))                .build();    }*/    @Override    protected void onCreate(Bundle savedInstanceState) {        super.onCreate(savedInstanceState);        setContentView(R.layout.activity_main);        prevView = findViewById(R.id.viewFinder);        textView = findViewById(R.id.scan_button);        if(allPermissionsGranted()){            startCamera();        }else{            ActivityCompat.requestPermissions(this, REQUIRED_PERMISSIONS, REQUEST_CODE_PERMISSIONS);        }    }    private void startCamera() {        cameraProviderFuture = ProcessCameraProvider.getInstance(this);        cameraProviderFuture.addListener(new Runnable() {            @Override            public void run() {                try {                    ProcessCameraProvider cameraProvider = cameraProviderFuture.get();                    bindPreview(cameraProvider);                } catch (ExecutionException | InterruptedException e) {                    // No errors need to be handled for this Future.                    // This should never be reached.                }            }        }, ContextCompat.getMainExecutor(this));    }    void bindPreview(@NonNull ProcessCameraProvider cameraProvider) {        Preview preview = new Preview.Builder()                .build();        CameraSelector cameraSelector = new CameraSelector.Builder()                .requireLensFacing(CameraSelector.LENS_FACING_BACK)                .build();        preview.setSurfaceProvider(prevView.createSurfaceProvider());        ImageAnalysis imageAnalysis =                new ImageAnalysis.Builder()                        .setTargetResolution(new Size(1280, 720))                        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)                        .build();        imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this), new YourAnalyzer());        Camera camera = cameraProvider.bindToLifecycle((LifecycleOwner)this, cameraSelector, preview, imageAnalysis);    }    private boolean allPermissionsGranted() {        for(String permission: REQUIRED_PERMISSIONS){            if(ContextCompat.checkSelfPermission(this, permission) != PackageManager.PERMISSION_GRANTED){                return false;            }        }        return true;    }    @Override    public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {        if(requestCode == REQUEST_CODE_PERMISSIONS){            if(allPermissionsGranted()){                startCamera();            } else{                Toast.makeText(this, "Permissions not granted by the user.", Toast.LENGTH_SHORT).show();                this.finish();            }        }    }}

回答:

所以我解决了这个问题。当添加TensorFlow模型来帮助对象检测时,显然它必须包含元数据(这样,当你想调用“getLabels()”及其相关方法时,它实际上会返回一个标签。否则,它将返回空值并导致错误。MLKit屏幕显示元数据要求

我使用并成功的模型是:mobilenet_v1_0.50_192_quantized_1_metadata_1.tflite

Related Posts

如何对SVC进行超参数调优?

已关闭。此问题需要更加聚焦。目前不接受回答。 想要改进…

如何在初始训练后向模型添加训练数据?

我想在我的scikit-learn模型已经训练完成后再…

使用Google Cloud Function并行运行带有不同用户参数的相同训练作业

我正在寻找一种方法来并行运行带有不同用户参数的相同训练…

加载Keras模型,TypeError: ‘module’ object is not callable

我已经在StackOverflow上搜索并阅读了文档,…

在计算KNN填补方法中特定列中NaN值的”距离平均值”时

当我从头开始实现KNN填补方法来处理缺失数据时,我遇到…

使用巨大的S3 CSV文件或直接从预处理的关系型或NoSQL数据库获取数据的机器学习训练/测试工作

已关闭。此问题需要更多细节或更清晰的说明。目前不接受回…

发表回复

您的邮箱地址不会被公开。 必填项已用 * 标注