Part I: DocumentationThe Documentation has been completed here. Part II: QuickMatch RuntimeWith 3 agents running (6 ROS nodes since each agent has a feature node and a quickmatch node) and each agent processing 6 images of size 1008 x 756, the QuickMatch Runtime is as follows:
The total end-to-end runtime (from starting reading image to finishing QuickMatch) is as follows:
Part III: Dealing with video processingIf we are dealing with a recorded videos, we can easily convert videos to images based on a certain frame rate, and then run the existing algorithm on the extracted images. I've found tutorials of how to do that in openCV (https://medium.com/@iKhushPatel/convert-video-to-images-images-to-video-using-opencv-python-db27a128a481) so this is definitely possible. If we are dealing with live videos, there's a ROS package called video_stream_opencv that enables a camera to publish a stream of ROS images. Here is the official ROS page and here is the GitHub page. QuestionI have roughly looked over Zack's NetMatch code (QuickMatch with contested points), but I have not looked deeply into it. It seems like there is a good amount of code that I need to read in order to understand what exactly is going on. Given the amount of time left this semester, I feel that I would not have enough time to finish implementing it in the distributed version and debugging.
I will work until the last day of final exam (May 9), which means I have two more weeks to work on things. So the question is: Should I work on a version with contested points or a version of analyzing recorded video frames? I am more confident that I will have the recorded video frames incorporated into the existing code in the next 2 weeks, but I will work on the contested points if that is of higher priority.
0 Comments
Part 1: Debugging FinishedAll the debugging has been finished. The current code works correctly. Regarding the bug from last week, I have added another field in the custom ROS message to transmit keypoints extracted during feature extraction. Since ROS does not support object publishing, I extraced [float x, float y, float size] from the 'cv2.keypoint' object and published them as an array of floats for each feature. They are transmitted from the original node where features are extracted, all the way to quickmatch_node where the graphing occurs after quickmatch algorithm is run. The 'cv2.keypoint' objects are rebuilt using the received data in the 'quickmatch_node.py', and the objects are using for graphing. Part 2: Result comparison with centralized codeThe following screenshots show that the result of the distributed QuickMatch is exactly the same as the result of the centralized QuickMatch (Figure 1 - 6). Figure 1: Centralized result (above) and distributed result (below) on image 1. Figure 2: Centralized result (above) and distributed result (below) on image 1 and 2. Figure 3: Centralized result (above) and distributed result (below) on image 1 and 3. Figure 4: Centralized result (above) and distributed result (below) on image 1 and 4. Figure 5: Centralized result (above) and distributed result (below) on image 1 and 5. Figure 6: Centralized result (above) and distributed result (below) on image 1 and 6. Part 3: Distributed results on new images takenSince the results of the distributed version have matched the ones from the centralized version, I ran the distributed version on the 6 new images I took before I left the lab due to the pandemic. The results are shown below (Figure 7 - 12). Figure 7: Results from the distributed version on new image 1. Figure 8: Results from the distributed version on new image 1 and 2. Figure 9: Results from the distributed version on new image 1 and 3. Figure 10: Results from the distributed version on new image 1 and 4. Figure 11: Results from the distributed version on new image 1 and 5. Figure 12: Results from the distributed version on new image 1 and 6. Part 4: Distributed results on new images taken with true k-meansIn order to debug, k-means has been set to the mean feature vector for node0 and negative numbers for all the other nodes. As a result, all the above results are generated with only node0 doing all the processing. In order to show the trully distributed match results, I reversed back the k-means to have features partitioned to different nodes for QuickMatch. Currently, 3 nodes and 3 quickmatch nodes are initiated, with each pair of nodes representing a robot (therefore a total of 3 robots). The results are shown in this video: https://drive.google.com/open?id=15jJ97EmTyEOgKRVhlUpxcqnZzmVeug6p The same 6 images are repeated 3 times. During each repetition, the matches done for one specific pair of nodes (one robot) will be shown. It can be seen that the matches have been distributed across all nodes. Part 5: DocumentationDocumentation has been created here:
https://docs.google.com/document/d/1KD0Jc04j5ioy37Hnn30I92voBEUOVhij1nFr4NVViA4/edit?usp=sharing I am currently working on it. The goal is to finish all the documentations by next week. Part I: Technical difficulty connecting via TeamviewerCould not connect to lab computer via TeamViewer multiple times this week, but it is eventually fixed with Zach by following instructions on this page: https://www.thewindowsclub.com/teamviewer-stuck-on-initializing-display-parameters Part II: DebuggingAs suggested by Professor Tron, the k-means centers were manually set to the mean vector of all features for node0, and set to negative values for all the other nodes. During debugging, it has been verified that all the features would go to node0 so that I could compare my matching result to the result of the centralized version. Bug 1: After numerous print statements, I found that the bandwidth calculated in 'calc_density' function is incorrect. The function was outputting [0 1 2 3 4 ... 2999] instead of the correct bandwidth. When implementing the function, I changed some code to deal with another error I encountered, and I forgot to change them back after the previous debugging. After I changed the code back to match the centralized version, the function outputted the correct bandwidth. Bug 2: In order to calculate matches, we have to keep track of which image a particular feature belongs to. There was an mistake in the implementation: I was keeping track of which node a feature belongs to, instead of which image. This bug resulted in all the features belong to the same cluster in the 'break_merge_tree' function, since all the features belong to node0 in my debugging process. After correction, all the image numbers from 0 to 5 appeared (since there are 6 images) in 'break_merge_trees". Bug 3: This mistake is similar to bug 2. In the 'features_to_Dmatch' function, I compared indices with the node numbers instead of image numbers for each feature. It took me a long time to find it, but after I corrected it, the number of matches the algorithm found is exactly the same as the number of matches the centralized algorithms found (Figure 1 & 2). Suspected Bug 4: (This is unresolved! Help!) After correcting bug 3, we can see that our matches are now correct. However, the matches drawn in the images do not look the same as ones in the centralized version (Figure 3 & 4). I believe this is due to 'quickmatch_node.py' re-calculating keypoints of the images for graphing. In the centralized version, the keypoints were calculated during feature extraction, and the same keypoints were used for graphing in the 'features_to_Dmatch' function. However, I was unable to use the same keypoints recorded in the feature extraction, because my feature extraction function and my 'features_to_Dmatch'/graphing function are in two different python files ('node.py' and 'quickmatch_node.py' respectively). As a result, I computed the keypoints again using the built-in `cv2.xfeatures2d.SIFT_create` function and `sift.detectAndCompute` function. I have discovered that the built-in feature extraction function does not output the same keypoints every time. For the centralized code, when I print out the keypoints, then results from one run would be different from results form another run. Although the keypoints were not used in match calculations, they were used for graphing. I tried to publish the keypoints obtained during feature extraction 'node.py', so 'quickmatch_node.py' can get the same keypoints for graphing. However, they are of the type <cv2.KeyPoint>, and ROS does not support publishing a cv2 object. I highly suspect that this is reason why the graphs are not showing up correctly even though the matches are already correct. Figure 3: Correct graphing of distributed version on the same image. Figure 4: Incorrect graphing results of the distributed version (above) and correct graphing results of the centralized version (below). Next steps:
Simple Distributed QuickMatch has been implemented (without points being contested). However, there are two problems with the current code. Problem 1: QuickMatch results are incorrectTThe distributed QuickMatch failed to find the matches even between the same images. This screenshot below shows the result of the centralized QuickMatch algorithm run on the same image: This screenshot below shows the result of the current distributed QuickMatch run on the same image: We can see that although the number of features are about the same, the distributed QuickMatch has not enough matches. The goal for next week is to investigate and try to make the result as close to the centralized version ans possible. My assumption is that there is a bug with my code. Part II: ROS constraintsMy distributed QuickMatch code crashes when running on big images. I initially used original images taken with my smart phone of 4032 x 3024 pixels (11MB), and the code would give an error saying insufficient memory. However, after I resized the images to 1/4 of the original size (1008 x 756 pixels), the error went away. I suspect that using multiple nodes in ROS to do image process at the same time has very high memory requirements.
|
Archives
May 2020
Categories |