Writely: Iteration 2

Outline of the content

  1. Things we know from Iteration 1
  2. Creation of Letters using SVG Images and Fonts
  3. Implementation of Active force feedback
    3.1. Guiding Force Feedback
    3.2. Guiding Force Feedback with Viscosity
  4. Implementation of Passive Force feedback
    4.1. Viscosity
    4.2. Vibration
  5. Experiments with Sounds while Writing Letters
  6. Addition of Metrics in UI
  7. Pilot Study with Friends
  8. Contribution Made
  9. Challenges Faced
  10. Conclusion
  11. Work for Future Iteration

Things we know from Iteration 1

  1. After going through multiple design iterations we have already created a working UI for Writely.
  2. We have figured out the forces we want to implement for our application
    - Full Feedback: It provides strong active feedback to the users. This will be similar to the force feedback of a solid virtual wall.
    - Partial Feedback: It provides partial feedback to the users to help them keep in their track.
    -Anti-Guiding: It forces users in the opposite direction of the actual path.
    - Disturbance: It provides small random forces in the direction of the user’s path.
  3. Further the forces can be divided into active and passive force feedback. The explanation of these feedbacks is done in ElasticVR, Passive training with upper extremity exoskeleton and Visuomotor learning by passive motor experience.
  4. We found that different letter have different forms of complexity. Hand-writing Rehabilitation paper has divided letters into curved and straight lines. In Investigating Haptic Guidance paper, letters in English alphabet have been divided into different categories to apply different forces.

2. Building Letters using SVG Images and Fonts

There are different ways to create letters and objects in Processing, such as by using PShape, Lines, and textFont. But the Visual structure created from these classes are difficult to reuse, takes effort and non of these Java Classes are part of Fisica so it’s not possible to implement methods available in Fisica (istouching, setForce, setFriction, setDamping, etc) into object made from these Classes. That’s why my first step in Iteration 2 was to create Fisica objects for letters using reusable structures.

2.1 Letters using SVG Images

There are multiple ways of creating letters using SVG images. The breakdown of the letter can be done based on its structure and its shape. Figure 1 and 2 shows how the Final SVG image was created using two and three SVG files respectively. For Iteration 2, experiment with different letter structure was done using Adobe Illustrator. Here the Figure 1 structure was created to implement Guiding force while the Figure 2 was created for normal disturbance and viscosity related force feedback (this is explained more in Section 3 and 4).

Figure 1: Letter A created using three different SVG images (created using Illustrator)
Figure 2: Letter B created with using two different SVG images (created using Illustrator)
Figure 3: Shows the printed X and Y points after SVG image is converted into FPoint

2.1 Letters using Fonts

TTF is an outline font standard created by Apple and Microsoft in the 1980s and currently is the most famous font file. TTFs include all of the needful alphabet, symbols and numbers for screen viewing and printing right in a single file. For our Writely application, to allow us to directly create the shapes of letters, I used multiple steps to convert the font file with extension TTF (TrueType Font) into FPoly Objects using the Geomerative library. The coding part of the conversion of TTF to FPoly objects is done in a separate Class called Alphabet using the method shown in Figure 4.

Figure 4: “create” method to convert TTF into FPoly objects
Figure 5: Output FPoly Object created for letter M and A
Video 1: Demo with font arialbd.ttf
Video 2: Demo with font Pacifico.ttf

3. Implementation of Active force feedback

The active force feedback works by constantly providing feedback to the user. In Writely application, concepts of narrow Wall-like effect and viscosity were used to provide active feedback to the user while writing letters.

3.1. Guiding Force Feedback

Figure 6: Narrow walls like structure for letter A
VIdeo 3: Demo for guided feedback

3.2. Force Feedback with Disturbance

In this step, letters were created using the TTF font file and then viscosity was added inside the letter area to create an effect of disturbance. The code used to add the disturbance is shown in Figure 7 and the demo of the force feedback effects is shown in Video 4.

Figure 7: Code for Force Feedback with Disturbance
Video 4: Force feedback with disturbance

4. Implementation of Passive Force feedback

Passive force feedback works by providing feedback to the user only when required. In Writely application, passive force works by gently providing vibration / dampening effects outside the boundary of the letters. The force feedback effect is added to nudge the user towards the area where the pantograph works normally.

4.1. Feedback with Vibration

To create this passive force feedback, I added a certain degree of vibration outside the boundary of the letter. This vibration force feedback was created by changing the FEE values randomly between 1 and -1 and by adding some dampening effects in the background. The real time demo of the feedback is shown in Video 5.

Figure 8: Code for Passive force Feedback with Vibration
Video 5: Demo with passive vibration feedback

4.2. Feedback with high Viscosity

To create this passive force feedback, I added a high degree of dampening effects outside the boundary of the letter. This viscous force feedback was created by adding dampening effects in the background. The real time demo of the feedback is shown in Video 6.

Figure 9: Code for Passive force Feedback with Viscosity
Video 6: Demo with passive viscosity feedback

5. Experiments with Sounds while Writing Letters

Sound plays an important part in creating an immersive experience to the user. A haptic multimedia handwriting and Haptics in teaching handwriting paper shows the usefulness of sound and haptics while writing letters. So in Iteration 2, I experimented with two different audio feedback to create an immersive experience for the users.

Figure 10: Code for change in frequency using Beads Library
Video 7: Change in sound frequency relative with the target (blue box) distance
Figure 11: Code for addition of sound clip using Minim Library
Video 8: Addition of sound near the edge of the letter

6. Addition of Metrics in the UI

Figure 12: Metrics added in the UI
Figure13: Class Calculator to measure metrics Speed, Velocity and Jerk
Video 9: Demo of change in metrics

7. Pilot Study with Friends

Figure 14: One of my friend using the Writely Application
  • All the participants were more interested in guided force feedback compared to disturbance. Participants showed positive reaction towards quick and high quality response from the guided force compared to the disturbance. One participant suggested that it would be better if the user had controls to change the force.
  • Most of the participants were interested in the passive force feedback with vibration. Also 2 of the people didn’t completely understand the implementation of disturbance.
  • All the participants were interested in the sound feedback integrated in the system.
  • 2 participants mentioned that they didn’t find the sound picked for the experiment pleasant.
  • 1 participant found the real time metrics useful and interactive while the rest of them saw those metrics as distraction while writing.

8. Contribution Made

  1. I worked on the implementation part of transforming TTF and SVG images into Fisica objects.
  2. Added metrics (velocity, acceleration and jerk) in the UI to calculate the cursor position.
  3. Added passive force feedback (vibration and viscosity).
  4. Added active force feedback (guided force and disturbance).
  5. Tested letter formation and position with different fonts style in the application.
  6. Integrated different sounds while writing letters.
  7. I conducted a small Pilot study for different force feedback and auditory feedback.

9. Challenges Faced

  • I received my Haply about three weeks ago and within those weeks I had to assembly my Haply, configure its software and learn a new programming language (Processing). So it was extremely difficult to work and contribute to the project during this short duration of time.
  • Working collaboratively in GitHub using completely new programming language has its own form of complexity like project structure, naming conventions, file formats, code indentation, git Ignore files and more. Also lot of our group time was spent going through GitHub pull requests, writing meaningful comments and understanding the code written by the other members of the group.
  • I had to spend a lot of time going through the Documentation for the Classes available in different Libraries. For example, I had to spend more than 2–3 day going through the Classes available in Fisica and Geomerative library to find the possibility of whether fonts / svg to FPoly is achievable. Similar cases were there for Minim and Beads libraries.
  • It usually takes considerable effort and time to resolve problems in Processing as one needs to figure out the solution by themselves. This is because there is less contribution for the programming language and solution cannot be found in websites like stack overflow.
  • It was difficult to implement the desired force feedback as we are still new to the Programming language and are restricted by the technical limitation of Haply.
  • In the Pilot study, all the participants were new to the force feedback and motor skill exercise. So it required a bit of time to explain to them about what I was trying to accomplish.

10. Conclusion

In the second iteration, I was finally able to work with Haply and practically carry out the programming and other technical parts of the project. Our initial UI built in first iteration provided a valuable guideline while adding different feedback and audio responses. Besides this, the pilot study was extremely helpful to understand the user perspective and identify the problems while implementing different feedback.

11. Work for Future Iteration

  1. According to the responses we got from the Pilot study, we will try to refine the implementation of force feedback and audio responses to provide more immersive experience to the user.
  2. Explore different quantitative methods and metrics to evaluate the effectiveness of Writely as a motor skill exercise.
  3. Explore ways to implement other different active and passive force feedback like Anti — Guiding Force Feedback.
  4. Some of the features included in our initial design like Alphabet Level, Haptic toggle button and instructions are still in progress. So in our next iteration we will work on the remaining features to be included in the Writely UI.
  5. We now have a better understanding of what we want to achieve from our project and How we plan to approach our experiments. So will also start with our initial draft for Introduction, motivation and literature section for our final paper.
  6. During the Pilot study, participants didn't have knowledge in haptics and motor skill exercise so for each person I had to explain about these terminologies and give examples. So for our next user study, we will create a standard script consisting of useful information about the experiment and haptic metaphors to help participants easily understand the project.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store