Design Document
EmojiHub allows users to interact by posting text content and reacting to posts using emojis, derived from their uploaded short videos. This guide provides an architectural overview, elucidating the interaction between the app's frontend and backend components and how the testing will be done.
2023-10-08 Design Document Created
2023-10-30 Change system architecture image
2023-11-08 Updated Frontend APIs and Backend APIs descriptions
2023-11-09 Edit layout of API tables more readable
2023-12-03 Add contents in API tables
2023-12-07 Add contents in Backend APIs & Edit Backend Class Diagram/DB schema
2023-12-09 Add Reaction related contents in FrontEnd API
Front-end of our app is structured with Jetpack Compose using Kotlin, pytorch, Retrofit and the back end is composed with Firebase and Springboot. We have fine-tuned PyTorch's X3D model to recommend the emoji that best represents the video, and the inference is performed on-device.
Subsequently, we focused on the performance of broad classification tasks and deployed using OpenAI's CLIP model through the Hugging Face inference endpoint. We believe that our first architecture using the X3D model presented a vision for on-device machine learning. Additionally, our second architecture utilizing the CLIP model enabled more extensive emoji inference and showcased the performance of zero-shot image classification models.
The following class diagram represents the relationships and dependencies between various activities and their functions that make up the app.
The backend class diagram includes 12 entities, their attributes and functions
The entities are:
- User Controller, Service, Dao
- Emoji Controller, Service, Dao
- Reaction Controller, Service, Dao
- Post Controller, Service, Dao
The DB model includes four entities and outlines their relationship. (All relationships are one-to-many).
The entities are Users, Emojis, Reactions and Post
- Role : Acts as our core development base where the app is constructed and managed.
-
Technologies:
- Android Jetpack: Manages UI elements, navigation, and lifecycle operations.
- Retrofit: Facilitates API calls to backend services, enabling smooth communication and data transfer.
- Role : Analyzes user-uploaded short videos, generating the most relevant emoji for representation and reaction purposes.
-
Technologies:
- PyTorch Mobile : A PyTorch library enabling the on-device execution of ML models, ensuring fast inferences and local data privacy.
- PyTorch X3D Model : A pre-trained model optimized for video understanding, aiding in the real-time prediction of emojis based on short user-uploaded videos.
| Class | Method | Parameter | return | Description |
|---|---|---|---|---|
| UserViewModel | fetchUser() | userId : String | UserDto | Fetch user information that matches userID. |
| UserViewModel | login() | userId : String password : String |
Authenticates a user with their userID and password. | |
| UserViewModel | registerUser() | userID: String password: String email: String |
Registers a new user account. | |
| UserViewModel | logout() | Expire authentication. | ||
| UserViewModel | signout() | Delete account. | ||
| EmojiViewModel | createEmoji() | videoUri: Uri | Pair<emojiName, emojiUnicode> | Creates a new dynamic emoji from a video. |
| EmojiViewModel | uploadEmoji() | emojiUnicode: String emojiLabel: String videoFile: File |
Boolean | Uploades a new dynamic emoji with a video file. |
| EmojiViewModel | saveEmoji() | id: String | Saves the dynamic emoji and its associated video. | |
| EmojiViewModel | unSaveEmoji() | id: String | Unsaves the dynamic emoji and its associated video. | |
| EmojiViewModel | fetchEmojiList() | Get list of all emojis. | ||
| EmojiViewModel | fetchMyCreatedEmojiList() | Get list of emojis which the current user created. | ||
| EmojiViewModel | fetchMySavedEmojiList() | Get list of emojis which the current user saved. | ||
| PostViewModel | uploadPost() | content: String userId: String |
Uploads a new post. | |
| PostViewModel | editPost() | content: String userId: String postId: String |
Edit post content. | |
| PostViewModel | deletePost() | userId: String postId: String |
Delete post. | |
| PostViewModel | getPostWithId() | id: String | Get the post. | |
| PostViewModel | fetchPostList() | Get list of all posts. | ||
| PostViewModel | fetchMyPostList() | Get list of posts of the current user. | ||
| ReactionViewModel | fetchReactionList() | postId: String emojiUnicode: String | Get list of reactions on a post | |
| ReactionViewModel | uploadReaction() | postId: String emojiId: String | Boolean | Add a reaction to a post |
| ReactionViewModel | getReactionWithId() | reactionId: String | Get the reaction with the specified Id | |
| ReactionViewModel | deleteReaction() | reactionId: String | Delete the reaction with the specified Id |
- Role : Secure User Management and Dynamic Data Handling
-
Technologies:
- Firebase Authentication: Manages user authentication, ensuring secure access and session management.
- Firebase Storage: Stashes user-uploaded videos and their corresponding recommended emojis.
- Firestore: Handles metadata storage, including user profiles, posts, and potential additional data regarding user interactions.
- Role : Facilitating backend logic, API management, and enhancing the coordination between Firestore and frontend.
| Model | API | GET | POST | PUT | PATCH | DELETE |
|---|---|---|---|---|---|---|
| User | /api/user | view all users | ||||
| /api/user/me | view the current user | |||||
| /api/user/signup | create new user | |||||
| /api/user/login | log in | |||||
| /api/user/logout | log out | |||||
| /api/user/signout | delete account | |||||
| Emoji | /api/emoji | view all emojis | create a new custom dynamic emoji | |||
| /api/emoji/:emoji_id | view a specified emoji | delete a specified emoji | ||||
| /api/emoji/save/:emoji_id | save emoji to his collection | |||||
| /api/emoji/unsave/:emoji_id | remove saved emoji from his collection | |||||
| /api/emoji/me/created | view emojis the current user created | |||||
| /api/emoji/me/saved | view emojis the current user saved | |||||
| Post | /api/post | view all posts | create a new post | |||
| /api/post/me | view the current user's posts | |||||
| /api/post/:post_id | view a specified post | update a specified post | delete a specified post | |||
| Reaction | /api/reaction/:post_id | add a new reaction to post | ||||
| /api/reaction/:reaction_id | view a specified reaction | remove a specified reaction from a post |
- Upon login or sign-up, user credentials are authenticated by Firebase Authentication via API calls.
- Users create posts (simple text) that are stored in Firestore via Retrofit, becoming accessible to other users for interaction.
- When reacting to a post, users upload a short video, and processed by PyTorch Mobile with X3D Model, then recommends an emoji by on-device ML.
- The short video is stored into Firebase storage and the reference to the video in Firebase storage along with the recommended emoji is also stored.
- Retrieves posts and related data (e.g., video URLs, emojis) from Firestore using Retrofit.
- When users view posts and reactions, data (posts, reactions, dynamic emojis) is retrieved from Firestore and Firebase Storage, displayed interactively through the UI powered by Android Jetpack.
- Users can choose or upload a video for a reaction.
- If users upload a video, X3D predicts the corresponding emoji, making users to react to a post with recommended emoji - short video pair.
- If users choose to use a dynamic emoji created before, users can select a dynamic emoji from their created dynamic emoji list.
| Iteration# | Test Type | Related Frameworks/Tools | Coverage Goal | |
|---|---|---|---|---|
| 1 | iter. 2 | Unit Test | Spring Boot, Firebase Firestore, Firebase Authentication |
User registration (Input validation, Password encryption, userID duplicate checking, user account information storage) |
| 2 | iter. 2 | Unit Test | Spring Boot, Firebase Firestore, Firebase Authentication |
Log In (login credential validation, password verification, token-based authentication) |
| 3 | iter. 2 | Unit Test | Spring Boot, Firebase Firestore, Firebase Authentication |
Log Out (invalidate user’s session / token on server-side) |
| 4 | iter. 2 | Unit Test | Spring Boot, Firebase Firestore, Firebase Authentication |
Account Deletion (user confirmation, data deletion from database, invalidate any remaining sessions or tokens) |
| 5 | iter. 2 | Unit Test | Kotlin, Android Jetpack | Log In page (containing button that maps with User registration page) |
| 6 | iter. 2 | Unit Test | Kotlin, Android Jetpack | User registration page |
| 7 | iter. 2 | Unit Test | Kotlin, Android Jetpack | Profile tab with Log Out and Account Deletion buttons. |
| 8 | iter. 2 | Integration Test | Kotlin, Android Jetpack, Spring Boot, Firebase Firestore, Firebase Authentication |
User Authentication using DB server |
| 9 | iter. 3 | Unit Test | Spring Boot, Firebase Firestore, Firebase Authentication |
Add post creation functionality |
| 10 | iter. 3 | Unit Test | Spring Boot, Firebase Firestore, Firebase Authentication |
Add post editing functionality |
| 11 | iter. 3 | Unit Test | Kotlin, Android Jetpack | Frontend functionality of Feed tab - listing posts, create posts, edit posts |
| 12 | iter. 3 | Integratiion Test | Kotlin, Android Jetpack, Spring Boot, Firebase Firestore, Firebase Authentication |
Create, edit, browse posts in Feed tab with several users using DB server. |
| 13 | iter. 3 | Unit Test | Kotlin, Android Jetpack | Frontend functionality of Profile tab - liked emoji collection, created emoji collection |
| 14 | iter. 3 | Unit Test | Pytorch | Tuning X3D model for emoji classification. Take a short video input and predict its emoji class in high reliability (human feedback based). |
| 15 | iter. 4 | Unit Test | Spring Boot, Firebase Storage, Firebase Firestore, Firebase Authentication |
Implement storaging video with infos(dynamic emoji) via Firebase Firestore and Firebase Storage. |
| 16 | iter. 4 | Unit Test | Kotlin, Android Jetpack | Frontend functionality of Emoji tab - upload video and recieve dynamic emoji result. |
| 17 | iter. 4 | Unit Test | Pytorch, TorchScript | Put torchscript style model to app assets, run classification prediction within app. |
| 18 | iter. 4 | Integration Test | Kotlin, Android Jetpack, Spring Boot, Firebase Storage, Firebase Firestore, Firebase Authentication, Pytorch, TorchScript |
Make customized dynamic emoji and store it to DB server. |
| 19 | iter. 4 | Unit Test | Kotlin, Android Jetpack | Frontend functionality of Feed tab - make a reaction on posts |
| 20 | iter. 4 | Unit Test | Spring Boot, Firebase Storage, Firebase Firestore, Firebase Authentication |
Handling reactions on posts in DB query perspective |
| 21 | iter. 4 | Integration Test | Kotlin, Android Jetpack, Spring Boot, Firebase Storage, Firebase Firestore, Firebase Authentication |
Make a reaction on post & see other users' reactions |
| 22 | iter. 5 | Unit Test | Spring Boot, Firebase Storage, Firebase Firestore, Firebase Authentication |
Implement API query for Emoji-Hub, Profile tab |
| 23 | iter. 5 | Unit Test | Kotlin, Android Jetpack | Frontend functionality of Emoji, Profile tab - listing emojis, save emojis |
| 24 | iter. 5 | Integration Test | Kotlin, Android Jetpack, Spring Boot, Firebase Storage, Firebase Firestore, Firebase Authentication |
Emoji listing test using DB server |
| 25 | iter. 5 | Integration Test | Kotlin, Android Jetpack, Spring Boot, Firebase Storage, Firebase Firestore, Firebase Authentication, Pytorch, TorchScript |
Full Test |
| Related Features | User Acceptance Criteria | |
|---|---|---|
| 1 | Make Customized Dynamic Emoji Share Dynamic Emojis through Emoji-Hub |
Given that a user is registered, when he is in Emoji tab and he presses New button and upload his short video from his local repository, then he gets dynamic emoji about the video and he can savt it to his collection, which is visible in Profile tab. And he can also save it to Emoji-Hub to share his dynamic emojis to other users. |
| 2 | Save Dynamic Emojis with Emoji-Hub |
Given that a user is registered, when he is in Emoji tab of the app and he presses save button of an dynamic emoji, then the user can save the dynamic emoji to his collection so that he can use it for reacting to other users’ posts at Feed Tab. |
| 3 | React with Dynamic Emojis |
Given that a user is registered, whenhe is in Feed tab of the app and he presses add-emoji-like-icon and select a dynamic emoji to be shared from his collection, then the reaction is applied to the post and it is visible to registered users and unregistered users with preview mode. |
| 4 | See Reactions for each Post |
Given that a user is registered or unregistered with preview mode, when he is in Feed tab and make a long tap on a post, then post related reactions page will appear in bottom up way. |
| 5 | See Ractions for each Post Dynamic Emoji Classification |
Given that a user is registered or unregistered with preview mode and get a reaction page of a post(as Acceptance Test #4), when he presses specific emoji class in header, then reactions classified to the emoji only will be displayed. |