Main content

Alert message

Voice Navigation (aka Talking Brick)

"Voice Navigation" turned out to be the very first enhancement to the MINDSTORMS® environment requested by one of our blind students. Within seconds of picking up a demo robot at our very first robotics meeting, he asked, "How do I turn on the sound?". I totally mistook the question he asked, thinking he was asking how to turn on sound output from the NXT. This demo program didn't output sound, so I said "It doesn't play sounds." But that wasn't his question. "How do I turn it on and navigate the menus?" he asked more forcefully this time. "Well, you can't", I answered. "At least not yet".

With a bit of NXC programming, we ended up with a usable "voice navigation" system. It is far from what it could be, but much better than what we had. Our first attempt which worked well for practice and our first competition looked like this: 
d51 0053

The display above contains the following information:

  • Line 1 - TSBVI - "Texas School of the Blind and Visually Impaired"
  • Line 1 - 8004mV - 8.004 Volts. Battery Power indicator.
  • Line 3 - left - announce - pressing the left triangular gray button announces what program you are about to run
  • Line 4 - right - next - cycle to the next program in the set of programs are in memory ready to run. Programs are stored in a "ring", where the last program wraps back to the first program.
  • Line 5 - square - start - the orange square button is the program start button
  • Line 7 - Pgm: 1 blueQ - 1 is the current program number, and the one that will run if the "start" button is pressed. "blueQ" is the name of the program for the (sighted) mentors to read. The talking brick only speaks letters and numbers. In this case it would announce "ONE". The small memory (256 KB Flash Memory) on the NXT do not allow longer names to be recorded and stored there. You quickly run out of available memory. The newer EV3 will remove this restriction (16 MB Flash Memory plus and a microSDHC slot).

This worked well enough in practice and also reasonably well at our first qualifying tournament. It must have since we were co-champions at that tournament, but we did have some trouble hearing the program numbers being announced from the NXT. The NXT does not have a large speaker, and the sound can easily be drowned out by the crowd noise. After discussing this in the team post game analysis (our post mortem), we decided to add large numbers to the display which could be read by our vision impaired students.

That display is shown below: 
d51 0047

Operationally, it has the same button interface as version 1 of the voice navigation had. The display now is geared towards the visually impaired student and has a huge number or letter in the display. In the example above a "1". The program name is in the upper right corner, here "yelQ" (For those of you who know the &qout;Senior Solutions" game, this is for the "Yellow Quilt Mission".) The upper left hand corner also has the program number. It should match the large print display and is there for debugging purposes. The lower right corner has "8239" which is the battery state in milliVolts.

One other oddity remains in using this interface. It is easy to turn on, simply find and press the square (orange) button. But how do I turn it off? We found the only reliable way to do this is to have only one true program on the NXT, the voice navigation (or launcher program). Thus, to start a program from an NXT which is off, we simply press the square button, wait for the NXT startup sound, then press the square twice. This starts the voice navigation. Now you can select which program to run and start it.

When it comes to turning off the NXT you do this specific sequence:

  1. Press the (gray) rectangular button 6 or more times.
  2. Press the (orange) square button once
  3. Press the (gray) rectangular button twice
  4. Press the (orange) square button once

After the press of the square button in step (2), you may be off already. In that case, continuing with steps (3) and (4) won't matter. But, if you aren't off at step (2), continuing with steps (3) and (4) will turn the NXT off. It's a bit of a pain at first, but it is a easily learnable sequence that will become second nature very quickly.

 

Robot build

We started the season as a rookie team. Both coaches and all 5 students (eventually 7) had never competed in FLL. Most of us had never actually built a LEGO® MINDSTORMS® robot. As with any rookie team, we sought out a viable base robot design on the internet, and used this as our base robot. One team mentor worked with 2 to 3 students to build a copy of that base robot. The mentor interpreted the pictures of the design, and conveyed it to the students who actually built the robot.

Building this way was a slow process and at times very frustrating. An alternate strategy we should have used is "dual build." You need twice as many pieces, but it works well. The mentor builds a robot in parallel with the student. The mentor puts several pieces on the robot, tell the student what was done and allows the student to hold and interpret what was done via touch. Then the student mimics the changes on his or her robot.

So I would summarize the process as follows:

  1. team builds robot with mentor
  2. team designs, builds and tests attachments
  3. team develops strategy and designs/tests programs

Below are pictures of the robots we used for our first and second competition:
phighter
Our first robot "Phighter".

A driving design element for our first robot was the desire (on the part of the students) for it to climb the "stairs" in the Senior Solutions Challenge. We were able to do that, but only about 10-20 percent of the time. In our very first match as a team, we were able to successfully balance after the stair climb. It was a thrilling moment.

phighter too
Our second robot "Phighter Too".

After our first competition, we decided to redesign our robot to take on tasks more reliably. We did succeed in building a more capable robot, but it was a great stress level on the team. In the future, I would not recommend such a major undertaking again. I would advise practice, "tweaks" and more practice, but no rebuild.

A critical success factor for all FLL teams is precision in starting the robot on its "missions." This can be done by sight or with the use of a "jig." Most teams use sighted alignment techniques, but we obviously could not. Every team member started at least one mission per match, so we developed and used a standard starting position for our first robot.

We spent many hours on different types of alignment. Here is a picture of one of our better (early design) alignment jigs, was a bit fragile but pretty accurate. It was not very good if it dropped! It did lead us down the road to better designs and strategies.

early jig
Early alignment jig.

For "Phighter," we incorporated into the design a tailpiece and some pegs off to the right side for alignment. If you back the robot into the corner, you can feel when the robot is at the precise angle we used to start. For our missions, we aligned into the back corner and touching the black standoff on the right side of the robot. (see top view photo below) It looked like this:

phighter tailpiece
Phighter's alignment jig.

phighter tailpiece
Phighter's alignment jig (top view). 

Obviously, since we won (as co-champions) our qualifier, this jig performed quite well. It did however force us to stay within a few turns of the home base. To move farther from home base, you would need sensors to help realign your robot or use another technique like "bump turn alignment." We chose "bump turns."

A "bump turn" is simple driving maneuver that allows the robot to re-establish that it is "square" to the direction it intends to go. You drive out to a point. Back into a wall and push just a bit (timed not by degrees). The robot realigns. Then you proceed, usually across the field on a true heading. This opened up several missions to us that we could do much more reliably than the "stairs." It is definitely a good technique to incorporate into your design.

phighter too bumper
Phighter Too's bumper.

The next two photos show a bump turn in action:
Back into Wall Drive off straight

The left (or top) photo shows the robot backing into the wall and lining up. The right (or bottom) photo shows the robot driving straight across the field ready to take on another mission farther from the home base than possible without the bump turn.

This new "bumper" design had us back to basics on a starting jig. We went very simple and chose to implement a spacer. It could be used for going "north" or "east." This gave us two starting positions versus one for our previous techniques. It was simple and reliable. Align the robot right (or left) against the jig and back tight against the wall. It could also be rebuilt easily if dropped! The new jig looked like this:

Bumper jig Bumper jig
Bumper jig Bumper jig

We (the coaches) had hoped to get into sensors from a robot design and a programming perspective, but simply didn't have the time. This was our first year as coaches and we had a lot to do to form the team from scratch. The next season we compete, I hope to have a least one or two sensors in use!

Preparing for and going to a tournament

Tournaments

  • Core Values. Core values overlap a great deal with social skills and are necessary throughout the day at a tournament. In the Core Values judging room, judges present an extremely challenging task and observe how well team members plan and work together (or not) to tackle the task. Following about 5 minutes of work on the task, judges may ask the team members questions.
  • Project/Research. Students have 5 minutes to give a creative presentation of their research and proposed solution to the real-life problem addressed by the theme of the year. Following the presentation judges question them for 5 minutes. Any coach assistance may disqualify a team.
  • Robot Design. Students bring their robot to a panel of judges who ask them questions about how they designed the robot and how they programmed it.
  • The Game. Teams have three matches in which their robot has 2.5 minutes to perform tasks on a 4’ x 8’ table set up representing the real-world problems related to the theme of the year. The robot must be preprogrammed and only the team members can stand near the table with only two of them at the table.

Plan and prepare for to smooth social integration

  • To help sighted judges question individual team members. For the three formal judging sessions, a team member handed judges supporting documentation with a cover sheet displaying a picture of the team with each members labeled with their name. Below the picture were the words, “Dear Judges: Please call us by name if you want to ask one of us a question. Thanks! The Dot Bots”.
  • To help Robot Design judges understand the differences in coding language, the Robot Design judges packet included large print and braille versions of some of the code that the team members had written for the robot game tasks.
  • To help peers and adults from other teams start conversations with team members. The Dot Bots chose to wear name tags with their first names printed in large print.
  • To appear available and interested in other teams, Dot Bots left their portable electronic devices behind.
  • To initiate conversations with other teams, our team brought bookmarkers with the team name in print and braille. Between judging sessions they distributed these popular small gifts to other teams throughout the Pit area.
  • To interest others in conversation, the team brought braille writers, slate and stylus, and index cards to demonstrate braille and help peers write their names in braille.

Explicit instruction and regular practice on challenging social and blindness skills

At a tournament, teams are expected to be independent of direct help from their coaches. In the Core Values judging room, coaches are not even allowed to come in with the team. This was an excellent motivation for students to practice independence, teamwork, and social skills. Specifically, the Dot Bots addressed and practiced:

  • Conversational and Presentation Skills.
    • Turn taking in conversations and in answering judges’ questions.
    • Giving relevant information in answer to questions.
    • Referring a question to a more knowledgeable or not yet addressed team member.
  • How to travel as a team. The Dot Bots paired up within the team: the totally blind students with less developed travel skills walked sighted guide with the students who had more vision and better travel skills. The team practiced walking this way and entering, performing in, and leaving simulated judging rooms. At the meets, the more visual students spread themselves through the group so the team was able to stay together and followed a guide to each of their judging sessions just as the other teams did.

Notes from install of Notepad++ (circa 2012). Your mileage may vary. The two most important things in here are the "F6" save_compile_and_download hot key and the setting of the larger print size (28) for our VI students. I intend to update and add more detail to this at some point. If you are reading this and have questions, contact us and it will accelerate the update process.

## 2012-11-13 - add notepad++ to laptop (Dell)

download http://notepad-plus-plus.org/download/v6.2.html (or current version)

Run the installer

on the Choose Compents screen (check) "Use the old, obsolete and Monstrous icon

#Add plugin NppExec

Plugins > Plugin manager > (check) NppExec -- INSTALL;

restart notepad++

F6 (same as Plugins > NppExec)

in the "Execute Commands:" box enter the following

   NPP_SAVE "C:\Program Files (x86)\BricxCC\nbc.exe" "$(CURRENT_DIRECTORY)\launcher.nxc" -d -Z2

save_compile_and_download

 

Settings > Preferences > General Toolbar

  • (check) Hide TabBar
  • (un-check) reduce
  • (check) double click to close document

Settings > Preferences > Editing

  • Folder Margin Style (check) none
  • Multi-Editing Settings (check) enable

Settings > Preferences > MISC.

  • SMART highligting (check)
  • Highlighting is case sensitive (check)
  • show only filename in the title bar Clickable Link settings (un-check) enable

Settings > Style Configurator

  • font size: 28
  • (check) bold
  • (check) enable global font
  • (check) enable global font size
  • (check) enable global bold font style

Surprisingly, only one rule deviation was required for our team to compete in FLL. We needed a line oriented programming language that can work with the braille display so that blind students can program and large print so our visually impaired students can read the screen. The visual-only programming environment of LEGO® MINDSTORMS® (NXT-G) does not accommodate visual impairment with the exception of screen enlargement. However, even modest screen enlargement makes MINDSTORMS practically unusable for the visually impaired. We sought and received a deviation from FLL game designer Scott Evans to use a the line oriented programming language called NXC. It is a "C" like language that works well with our 80-cell braille displays and screen reader software (JAWS®) and also supports LEGO MINDSTORMS. In practice, we usually turned off JAWS and used the braille display only.


Coding without a Visual Language

Coding with the required visual programming environment (image below) and language NXT-G turned out to be impossible for our blind student and nearly so for all but one of our visually impaired students. One of our VI students had actually used NXT-G when he still had good vision, but as his sight was now such that it was very difficult for him to make any decent progress programming with screen magnifiers. Without screen magnification, he too could not use NXT-G.

Visual programming environment with icons for commands

Our Programming Environment

Programming with NXT-G was really not an option for us, even with our most visually capable student. We also had the goal to have everyone on the team to be able to do every role. This meant we had to choose an environment that was line oriented and worked with JAWS® instead of a visual environment. As we found over time, the best setup was using JAWS with the sound turned off and using an 80-cell braille display. In the attached picture, you will also see that we "disabled" the touchpad cursor control by covering it with an index card. Low tech, but effective.
programmer workstation, laptop with braille display

With the evidence in hand that we required a deviation from the rules on the programming environement, we went ahead and sought a deviation. Kay wrote a letter to Scott Evans, head FLL game designer at FIRST. Scott answered quickly and gave us clearance to go ahead and compete with whatever software environment worked for us. That turned out to be the Notepad++ and the NXC programming language.

We had a couple of choices of text oriented coding languages and settled on NXC. Other line oriented languages like RobotC would most likely work fine, but are not free. NXC is totally free. We made some initial tests of the environent, specifically with screen magnifiers and JAWS and decided to use NXC. Later, we introduced the braille displays, which also worked well, so we feel the choice of NXC was perfect for us.

There is a development environment for NXC called Bricx Command center which we tried and found cumbersome. We switched to Notepad++ which is a highly customizeable environment which worked well for us. The primary thing we did was to simplify the environment giving fewer bells and whistles for the kids to get confused on. We also used the Notepad++ macro capability to have a simple way of compiling and loading out code on the NXT. Here are some Notepad++ setup notes here.

We started using a JAWS workstation only without the use of the braille display. This turned out to be somewhat of a disaster. Most of the students were attempting to learn NXC, Notepad++ and programming all at the same time. Trying to instruct and have the students attempt to "find" their place in code was frustrating for them and me. It was definitely no fun and not a good learning environment. When we added the 80-cell braille display and turned off the sound from JAWS, things went extremely well. The students could be scanning through the program on the braille display while we talked and know exactly where they were. A neat feature of the braille display is also that once you know where you are, there are "Some important keys/key sequences:
"JAWS Key" - Ins-"J" - on desktop
Ins-F4 - unload JAWS
Ctrl-Alt-"J" - hot key JAWS
Ctrl - "quiet" JAWS key

JAWS Settings
> Braille General
Active Crsor follows braile (check)
Contracted braille (off)
> Advanced
Status cells (none)

hot insert" keys that allow direct insertion of characters at that point. Efficiency and confidence in coding went up dramatically.

NXC offers a larger set of commands and capbilities than is available in NXT-G. For fairness and to simplify the programming task for our new to programming students, we created a subroutine library which mimicked the coding blocks of NXT-G. We had commands like

Move, Right, Left, RightSpin, LeftSpin, ArmDown and ArmUp.

In our code, we use the prefix "tsb" (Texas School of the Blind) to avoid name space collisions, so Move was tsbMove, etc. A sample move command would be:

     tsbMove(50, 360, COAST);

This would use the builtin NXT encoders to move both wheels 360 degrees at 50% power and then COAST to a stop (instead of BRAKE).

Sample Programs

I'm going to include some sample programs here. They range from simple standalone programs that are accessed as normal through the NXT menus, to a single "chooser" program which will run a set of missions and give voice feedback to the user. All mission related programming per FLL rules must be done by the students and we followed that rule. Accomodation tasks, like the "Chooser program", the "include file" and the "roller program" were obviously written by the coach.

scottevans clean

Text of Screenshot above

-------- Original Message --------
Subject: RE: Blind students and FLL
Date: Tue, 31 Jul 2012 13:59:41 +0000
From: Scott Evans 
To: Kay Pruett 

Hi Kay,

Absolutely. I'm copying Jen O'Callaghan, who works with our Operation Partners in Texas, so she can spread the word that it shall simply be "understood" that these kids need to and are allowed to program under this exception to our Allowable Software rule.

Have an awesome day,

Scott


 

From: Kay Pruett 
Sent: Tuesday, July 31, 2012 9:51 AM
To: Scott Evans

Subject: Blind students and FLL

Scott,
I am a rookie FLL coach and a teacher at Texas School for the Blind where we are starting a team for the 2012 season.

Danny Diaz suggested that you might be able to help with a problem we have with my blind and visually impaired students accessing the mandated software. The Mindstorms software is not accessible by screenreader or braille display. We would like to get approval for using a (different) accessible software for those students who are not able to see well enough to use the visual Mindstorms programming tool. This would enable all of my students to experience programming the robot for themselves.

Thanks,
Kay

Kay Pruett
Texas School for the Blind and Visually Impaired

Authors : Kay Pruett (TSBVI) <> , Jim Allan (TSBVI) <>, Gerry Cocco, LASA Robotics coachThe DotBots - TSBVI Robotics Team

Poster Session, Saturday August 2, 2014 AERBVI conference, San Antonio, TX

It started as a question, "Could my kids do this???" and ended with a team of seven blind and visually impaired students (upper elementary and middle school students) competing on a level playing field in two FIRST Lego League (FLL) tournaments.

Events included Robot Performance, a Research Project and Presentation and a social skills exercise called Core Values.

Their co-championship team was the prior years Central Texas winner that advanced to the FLL World Championship.

Visually impaired students and teams will likely benefit from taking some specific actions to prepare for positive social interactions at a tournament. Tournaments have four formal areas of judging, but teams are also judged on overall performance throughout the day. When not in scheduled events, teams wait and interact with one another in the “Pit” (often a cafeteria).

The answer was absolutely "YES."


Pros and Cons

Why to Start a Blind and VI team:

  • phenomenal growth in social skills
  • improved academic skills
  • mobility improved
  • lots of fun
  • programming could lead to career
  • team becomes ambassadors with mainstream
  • show capabilities of blind/vi students in future world of robotics
  • opportunity for kids to make amazing growth
  • opportunity to change attitudes of future employers and coworkers
  • opportunity to interact with people with visual impairments on an equal footing
  • FIRST is a supportive and welcoming community
  • lots of training materials and resources exist
  • we have the VI accommodations sorted out on this site

Accommodations

Why NOT start a team

  • it takes dedication and time
  • it takes money - around $1000 (however, there are scholarships, grants and industry support available!)
  • training is available (see resources section)
  • need a space for a 4 foot x 8 foot - practice table
  • need a minimum of 3 kids, 5-6 is ideal, 10 is the maximum allowed 

Resources

FIRST® LEGO® League (FLL) has a large amount of online information on starting a team, running a team and taking a team to a competition. Our focus on these pages is to highlight the things you need to do above and beyond that for blind and visually impaired (VI) students. The links given here in "Resources" will point you to a few of the important starting points to learn more about FLL:

Start a Team

Dot Bots - in the news