[IQP] ASSISTing Panamanian Education: Using the online learning tool, ASSISTments, to improve math education in Panama

Sponsor: Autoridad Nacional para la Innovacion Gubernamenta Assments
Student Team: Alexander J. Gikas
Jennifer L. Golden
Casey R. Kracinovich
Jose Antonio Spiegel
Abstract: Panamanian math education is weaker than similar neighboring countries. We conducted a trial with a local private school and surveyed/ interviewed students, teachers, and government officials to determine if incorporating educational technology, ASSISTments, into Panamanian schools would improve test scores, student engagement, and teaching efficiency. The test scores of students using ASSISTments improved, but there are many limitations that prevent us from drawing concrete conclusions. Therefore, we recommend a more extensive trial to validate these results.
Links: Final Report

Executive Summary

A 2010 survey found that 95% of Panamanian teachers consider math to be the subject that their students struggle with most (Valderrama Bahamóndez & Schmidt, 2010). Panamanian math education is weak, scoring poorly on international standardized tests both globally and as compared to neighboring countries with similar investment in education, such as Colombia (OECD  2009, LLECE 2014). Studies show that Panamanian math students struggle with basic skills such as the multiplication table and the four basic arithmetic operations (Valderrama Bahamóndez & Schmidt, 2010). Likely explanations for Panama’s problems with math education include socio-economic inequality, limited funding and resources, and government turnover.

Many educators are exploring educational technology as a potential solution to improve learning in the developing world. Large scale meta-studies demonstrate that under ideal conditions, educational technologies can have a positive impact on student performance in the developed world (Means et al., 2009, Cheung & Slavin, 2013). Many obstacles arise when implementing educational technologies in developing countries, such as limited equipment, misunderstandings of technology, and economic feasibility (Woolf, Arroyo, & Zualkernan, 2011). In order to be successful in Panama, educational technologies must be affordable, accessible, and practical.

ASSISTments is an educational website that specializes in math and has shown promising results in the United States. It helps students work through problems, and provides teachers with immediate feedback on student performance. ASSISTments was originally geared towards middle school math standardized test preparation, but it has now expanded to include other subjects such as statistics, science, and chemistry. Because ASSISTments is free and accessible on any device with internet access, it could be used in the developing world.

Project Goal, Research Questions, and Methodology

We worked alongside the Autoridad Nacional para la Innovacion Gubernamental (see Appendix A for more information) to determine if incorporating educational technology for math, such as
ASSISTments, into Panamanian schools will improve test scores, student engagement, and teaching efficiency. To accomplish our goal, we defined four research objectives:
1. Assess the state of middle school math education in Panama.
2. Assess Panamanian experiences and attitudes concerning educational technology.
3. Develop a feasible, efficient, and scientific trial to test ASSISTments in a Panamanian middle
school.
4. Analyze the results of the ASSISTments trial through both test scores and student/teacher
opinion.

To obtain information, opinions, and experiences on the state of math education and use of educational technologies in Panama, we conducted surveys and interviews. We surveyed middle school
students at a local private school, and math teachers across Panama. We interviewed government officials with the Ministry of Education and two local private school teachers. With the collaborative data and research on educational experiments, our team designed a trial to test ASSISTments in the Panamerican School. We worked with a seventh-grade class, splitting it into a control section and an experimental section. The experimental section used ASSISTments in class for short classworks on six days and a long class assignment on one day. The control section did identical assignments on paper. Before and after the trial, we interviewed the teachers and surveyed the students.

Findings and Analysis

Our findings are divided into four sections based on our objectives.

Objective 1: Assess the state of middle school math education in Panama.

Finding 1: Because most teachers, particularly in primary school, have received inadequate training, they generally teach using “old-fashioned” methods focusing on rote memorization, thus undermining students’ basic skills, retention levels, math applications and analysis skills, and test performance. Primary school teachers in Panama are only required to complete high school, with a specialized teaching diploma. Since primary school teachers are often not specially trained to teach math, students often do not learn math basics at a young age. Many of the people that we interviewed described Panamanian education as “old-fashioned.” Instructors teach in the same way that they were taught, with an emphasis on rote memorization rather than analysis and critical thinking. Ideally, ASSISTments and the feedback that it provides could help turn teaching into a two-way process. Also, ASSISTments hints could help students to work through analysis problems.

Finding 2: Most Panamanian students find math to be particularly difficult, as shown by our surveys and confirmed by a more widespread survey. 80% of the 100 middle school students that we surveyed rated the difficulty of math to be 3 or higher on a scale from 1 (very easy) to 5 (very difficult). Most of these students (56%) chose math as their most difficult subject. A more widespread survey found that 44% of Panamanian students find math to be their most difficult subject (Valderrama Bahamóndez & Schmidt, 2010).

Objective 2: Assess Panamanian experiences and attitudes concerning educational technology.

Finding 1: Some Panamanian public and private schools have attempted to implement educational technologies such as Smartboards, Balboa Laptops, and Destino Matematico, but most have ultimately failed because of insufficient teacher training and frustrations such as losing a significant amount of class time to setting up the technology. Panama has attempted to introduce several educational technologies such as Balboa Laptops, Smartboards, Schoology, Destino Matematico, and Khan Academy. From these experiences, four common problems arose:

  • Truncated programs: Due to government turnover, many programs are begun but not completed.
  • Overuse of class time: Classes in Panama are short, sometimes only 30-45 minutes. Teachers found that time taken for setting up computers and connecting to wireless internet for an online assignment could be up to half of the class time.
  • Insufficient training: Often, teachers did not receive enough training and thus did not fully understand the technology and how to efficiently use it.
  • Insufficient support: Programs instituted by the government often lacked significant support. As a result, teachers sometimes encountered frustrations such as technical bugs, and abandoned the technologies as a result. Understanding these experiences helped us avoid the same problems in the ASSISTments trial. We provided substantial teacher training before the trial and support throughout the trial to avoid making the same mistakes.

Finding 2: Of the 100 seventh- and eighth- grade private school students that we surveyed, nearly all of them would be willing to try an educational technology. Students appear to be enthusiastic towards using educational technology. Students who have already used educational technology likely had a positive experience with it because they are interested in using it again.

Objective 3: Develop a feasible, efficient, and scientific trial to test ASSISTments in a Panamanian middle school.

Finding 1: Rather than a true experimental design that is neither feasible nor ideal (due to randomized groups), a quasi-experimental design in which non-random control and experimental groups drawn from pre-existing classes are given identical pretests and posttests is the most feasible trial design. According to the quasi-experimental design, both groups are observed and tested prior to the start of the trial. The experimental group goes through the program, and the control group does not. Afterwards both are again given identical tests. The results of the tests can then be compared to show the effects of the program. The true experimental design is identical to the quasi-experimental design except that the control and experimental groups must be random. We had to use nonrandom preexisting classes as our control and experimental groups, so the true experimental design was not feasible for our trial.

Objective 4: Analyze the implementation of ASSISTments in schools, as shown through both test scores and student/teacher opinions.

Finding 1: After using ASSISTments, average student test scores were 10% better than the class average of all tests taken in the previous trimester, whereas the control section saw no significant improvement. The experimental section saw more improvement in grades than the control section. On the pretest, both sections performed the same, proving that they had identical prior knowledge on the topic. On the posttest, the experimental section averaged 4.1 (of 5), while the control section averaged only 3.6 (of 5). Without considering our limitations, it appears as though ASSISTments did show potential as a possible solution to increasing student math performance.

Finding 2: The teacher believes that ASSISTments increased his time efficiency and has such a strong positive opinion that he is already using it in his other class sections. The teacher believes that the program was not a large investment of time to learn and did increase the efficiency of his class. Because time efficiency was difficult to measure, we obtained his general opinions on ASSISTments as well. He thought that the ASSISTments feedback was valuable to teaching. He also thought that ASSISTments would be useful in any school that has the necessary facilities for it.

Finding 3: The majority of students believed that ASSISTments was a useful tool that increased their engagement in math class, and all of them thought that other students would like using it. Based on student opinions drawn from our survey, according to 86% of students, ASSISTments was more engaging than traditional paper classwork. Conversely, the teacher believed that students were no more interested and engaged in their classwork than normal when using the program. When asked to rank ASSISTments usefulness on a scale from 1 to 5 (most useful) 86% of students ranked ASSISTments at a 4 or 5. All students would recommend ASSISTments to students in other classes and schools.

Trial Analysis and Conclusions

While student test scores did improve, and both students and teachers had positive opinions on ASSISTments, many limitations make us question the validity of our trial. These are:

Class topic We only tested one class topic. It is entirely possible that ASSISTments could have been more or less useful for other topics. It is also possible that the experimental section grasped the concept of radicals more easily than the control section, regardless of ASSISTments. Testing a wider range of topics would better validate the effect of ASSISTments on grades.

  • Feedback/trial set-up The teacher understood the benefits of the ASSISTments feedback, but did not use the feedback, a major feature of ASSISTments, to its full potential. This was largely because the setup of the trial in his specific class did not allow for enough in-class reviews of ASSISTments assignments. Our trial was too short to be able to include more assignments, and thus more inclass review would allow for the feedback to be used to its full potential.
  • Change in routine Related to new methods of learning: Students may have been influenced by the change of their everyday learning routine. A new routine may have been more engaging, thus increasing student focus and learning. Related to our presence: Students may have been influenced by our presence in their classrooms. This may have affected their behavior, class performance, or their interest in ASSISTments.
  • Computers When talking to students after the trial, many said that they enjoyed using computers in classrooms. This led us to question whether their improved test scores were specific to ASSISTments. It is possible that the use of any computer program would have caught students’ attention, thus showing improvement in test scores and positive opinions.
  • Sample size Only one class of 24 students used ASSISTments, making our sample size small and uniform. We only worked with one teacher. Our results cannot be generalized for all Panamanian schools. Other students and teachers may have reacted differently to ASSISTments.

Many of these limitations were inevitable because of the two week time frame. We cannot say for sure that the increase in student test scores or the positive reception to ASSISTments was entirely due to ASSISTments and not one of these many other influences. Ultimately, our trial is inconclusive because of our various limitations.

Recommendations

To make ASSISTments an appealing program to more schools, especially those in the developing world, we recommend the website creators consider the following changes to the program: viii

  • General changes
    • A space for students to include their scratch work.
    • An option for weighted grading.
    • The ability for students to message/chat their teachers.
    • The option for a time limit for assignments to be completed.
    • A more user-friendly builder tab.
  • Specific to developing countries
    • The option of non-US time zones.
    • A version that is available offline.
    • The option for languages other than English.
    • Organize content by topics rather than U.S. Curriculum.

We recommend that the Panamanian government (AIG, SENACYT, or MEDUCA) replicate our trial, taking steps to avoid the variables that we encountered such as sample size limitations and time constraints. Because our trial showed improvement in student test scores, but had numerous limitations, we recommend that ASSISTments is tested more thoroughly with a similar trial to validate our findings, making the following extra considerations:

  • Choose teachers who vary in computer competence
  • Spend more time in class and lengthen the trial
  • Cover more topics
  • Use experimental and control class sections that are even in behavior, schedule, and academics
  • Increase the use of ASSISTments feedback through more in-class reviews on assignments

We recommend that the Panamanian Government (AIG, SENACYT, or MEDUCA) run a similar ASSISTments trial in public schools, taking into account both the problems that our trial encountered and the following special considerations for public schools:

  • Facilities may be limited: Public schools and public school students are far less likely to have sufficient access to computers and internet than the private school that we worked with.
  • Teachers may require more training: The teachers that we worked with were well educated, experienced with technology, and extremely enthusiastic about educational technology. As a result, teaching them how to use ASSISTments and how to incorporate it into their curriculum ran smoothly and quickly. It is possible that teachers in public schools may not have the same level of background knowledge and enthusiasm towards ASSISTments, so more thorough training may be necessary before they incorporate it into their classrooms.
  • The language barrier will be more of a factor: Though content can be written in Spanish, all of the pre-built content in ASSISTments is in English, along with the interface. Being a bilingual school, this was no problem for the Panamerican school, but it would likely be an obstacle for many public schools.

ASSISTments is not perfect, and other technologies may be more beneficial. It could be of great benefit to test different educational technologies that have different purposes to see their impact on Panamanian education. A few examples are:

  • Interactive educational game
  • Website where notes and videos can be posted
  • Messenger for student and teacher communication

We recommend that the Panamanian government (AIG, SENACYT, or MEDUCA) institutes a program to both train and support interested teachers with the necessary facilities who want to use technology in their classrooms. Educational technologies in Panama tend to fail because of insufficient training and frustrations. To combat this, the Panamanian government could run an educational technology program and invite teachers who have proper facilities. A trainer will work with teachers. After a teacher feels comfortable with the educational technology he or she can begin adapting the tool to his or her curriculum. After classroom implementation has begun, the trainer needs to give the teacher all the support required to address any problems.