Ep. 90: The Scientific Method

Fraser Cain: I was looking through Wikipedia today and there was an entry for Universe Today, there was an entry for me, there was an entry for you, there was an entry for Slacker Astronomy, but there didn’t seem to be an entry for Astronomy Cast.

Dr. Pamela Gay: No. Astronomy Cast is the long lost forgotten Wikipedia entry. Someone needs to write it.

Fraser: Now we can’t write it because that would be wrong. And for us to actually go on Wikipedia and write it would be frowned upon by the community we understand that. But perhaps some of the listeners out there who wanted to, you know, put some little bit of information here and there, that would be greatly appreciated. As well, we should get a lot of links from Wikipedia from various show topics.
 
But, if you wanted to look through some of the topics we’ve handled in the shows, and maybe make sure there were links to our shows, that would be awesome too. Once again, we’re not saying that you should, but that would be a great help.

Pamela: It would make me smile stupidly.

Fraser: Yeah, and, you know, if you have any questions for us to get some facts just us an e-mail at astronomycast.com. Alright, now you have heard me say this 90 times so far, how we know what we know…But, how do we know how we know what we know?
 
Astronomers like all scientists use the scientific method. And without the scientific method we would think the Earth was flat, a few thousand years old and the center of the Universe. But with the scientific method everything changes biology, chemistry, and physics. You can count the number of changes that have happened to human society because of our discoveries made with the scientific method.
 
But, you know for me anyway, I never really understood the scientific method until I did a lot of research on my own and at the university. I think we want to take some time and go into here on the show. So Pamela, can you give us a kind of a succinct description of the scientific method?

Pamela: Probably the briefest way to state it is if you have an idea and you go, I wonder how. Then you do a series of experiments or observations to test ‘how’ whatever that how may be and then rule out or sustain the answer to that I wonder how. Then you have participated in the scientific method.
So when your phone breaks and the first thing you do is you jiggle all the cables, you’ve asked the question, I wonder if the cables are the reason my phone doesn’t work. You have performed an experiment to see is that the reason, and the answer to that particular experiment was no.
 
As you work through your series of, Hmm, I wonder, you’re following through the scientific method at every step.

Fraser: That’s interesting. So, troubleshooting a technology problem is following the scientific method. Can you give kind of the specific steps then, that I guess that scientists follow when they go through the scientific method?

Pamela: Well the exact steps are going to depend a bit on what field you’re in. So, for instance, if I’m doing theoretical physics and I want to answer the question of, is it possible to think of a 37 dimensional particle that is consistent with our understanding of the Universe? I’m going to have a slightly different set of steps than if I start from the premise, if I stand on the moon do I weigh 1/6 of what I weigh here on the planet Earth?

Fraser: Or why doesn’t my phone work.

Pamela: [Laughter] Or why doesn’t my phone work. In general the first thing you do is to come up with what we call a hypothesis. This is an idea that you want to test. Is it true or not? Is the reason my phone isn’t working because the cables are loose?

Fraser: So would that be like a whole experiment? Like say I was going to write a research paper on why my phone doesn’t work. Would I write a paper, does my phone not work because the cable is loose?

Pamela: That would be a very lame paper.

Fraser: Of course, of course.

Pamela: [Laughter] Well with papers there are folks out there who do the least possible amount of work per journal article. But a good paper is one that says we had this issue, we tested this issue in 12 different ways so this would be my phone didn’t work. I tested the cable. I then made sure with the phone company that I had paid my bill. I then replaced the unit at that particular outlet with a different phone to make sure the outlet was working. You go through a whole series of different tests and you explain in your journal article all of the different tests you performed and what their outcomes were.

Fraser: Right, but in this situation you are bunching together a whole bunch of experiments all at the same time and then publishing the results. You could just as easily publish a whole paper just on why my cable isn’t the reason why my phone doesn’t work.

Pamela: And with really hard expensive cables that’s pretty much what happens. We’ve all heard the story at this point with the cosmic microwave background original detection by Penzias and Wilson. They had a giant radio telescope. It had this weird background hiss. They made the hypothesis that there were pigeon droppings inside their detector. They cleaned their detector to remove the pigeon droppings and discovered the cosmic microwave background.

Fraser: Alright, so let’s go back to the process then from initiation to completion. You’re saying they make a hypothesis.

Pamela: You then test the hypothesis. It has to be something that is reproducible so doing just one test generally isn’t sufficient. You do a series of tests to prove that whatever it is that you’re doing is reproducible. Then you try and come up with an explanation that draws together your, hmmm, I wonder…, with the results of your experiments into a coherent model that fits with our present understanding.
 
So, it has to hook into what was already understood without saying, well gravity doesn’t apply in this one particular scenario because well I’m just going to ignore it.

Fraser: Right, you’re getting at Einstein’s theory of relativity here, right.

Pamela: And, just about everything we do everyday. It’s amazing how often people forget gravity.

Fraser: Right, but Einstein had to come in and say here’s how gravity really functions but he had to include all of Newton’s calculations as well and then extend it further.

Pamela: Trying to figure out how things orbit the sun is perhaps one of the best examples of a theory starting simple and growing fabulously and still having room to grow more.
 
Kepler originally looked out of the planetary motions and tried all sorts of crazy things. He started with the hypothesis well, a circle is the perfect shape so they must all move on circles. That didn’t quite work. After a lot of mathematical testing and after looking through an amazing collection of data that was extremely precise, he came up with his model that planets move on ellipses around the sun. Some of the ellipses could be perfect circles, or they could be special types of ellipses like parabolas.
 
He had this great working model and it described accurately the motions of most of the planets. Mercury was a little bit screwed up. But it didn’t say why they did this. So Newton, who came along much later, came up with the theory of gravity. He was able to explain that the Moon moving around the Earth, the Earth and the Moon moving around the Sun, we’re just falling and failing to hit the surface. We’re falling and falling forward at a rate that keeps us from hitting the surface of the Sun. It keeps the Moon from hitting the surface of the Earth.
 
He built a very nice theory of gravity that was able to explain for instance, how the different planets affect one another’s orbits. So, when we found Uranus, its orbit and Neptune’s orbit matched together nicely because Neptune was getting deformed on its orbital trajectory by the extra pull from Uranus. But, Mercury was still screwed up.
 
Then Einstein, also looked at Newton’s theory’s that was built on Kepler’s theories, and said well, let’s take this a step further. Let’s look at this in the environment of extremely dense objects. Let’s look at this at extremely large velocities approaching the velocity of light.
 
And, he came up with his theory of relativity. Now even the theory of relativity breaks. It doesn’t explain what is going on inside of a black hole. It ends up coming up with infinities and singularities. I don’t think anyone actually thinks that a black hole is nothing more than a point in space.
 
But, recognizing this we know that someday some brilliant person or a team of people are going to come up with a theory of gravity that brings the theory of relativity and the theory of quantum mechanics together so they can work in a unified way. Perhaps using what we are now using the placeholder name, quantum gravity for.

Fraser: And once again, this theory will need to incorporate all of Newton’s material and Einstein’s material and then explain the things that Einstein wasn’t able to explain.

Pamela: So theories are capable of growing and building upon one another. And what forces them to grow on one another, is new observations. The realization is yeah, Mercury refuses to behave normally if you look at Kepler’s equations, or if you look at Newton’s equations, there’s got to be something else out there.
 
When we see something, we observe something, when we do an experiment, and the results don’t match what we expect, that’s when we have this really cool realization that we don’t understand something.
 
This is why scientists exist, to pick apart the things that nobody understands and find the little bit of new knowledge that is inside this intellectual knot that we’re trying to untangle.

Fraser: So, I want to go back to the process. We’ve got your hypothesis and then you make some mixed observations. You perform some experiments. Then you try to take the observations that you’ve made and try to fit these back in to try to create a model that matches your hypothesis.
 

Pamela: This is exactly what we do and we try to do this in a mathematically rigorous way or in at least a chemically rigorous way when we start dealing with biologicals and chemicals, so that we can explain things in detail. We can use statistics and in a lot of cases perhaps the majority of the cases, the things that we start out looking for, the things that we start trying to prove are true, turn out to be totally false.
 
This is what makes the scientific method so cool, I can sit around blabbing with a bunch of friends at 3 in the morning and say anything I want and claim that it’s true because I’m more charismatic then you are, or at least louder. But, without the data, numbers or evidence, all I am is a loud bit of hot air in the corner of the room in the middle of the night making claims.
 
With the scientific method, I have to stand up and say, here is a mathematically rigorous statistically proven carefully tested explanation of why I’m right. And sometimes you start off with a great 2 in the morning theory that you’ve decided is going to be your dissertation, be your master’s thesis. This is going to be the source for going after a huge amount of grant money.
 
You go through any different number of things that allow your career to move forward. You get part way through the project and realize, oh dear, I was so totally wrong. In these cool moments, we both think, wow I hopefully just prevented someone else from wasting a lot of time trying to do this thing that sounded so cool, because I’ve proven it doesn’t work. Um, and you cry.

Fraser: Well, I’ve seen something kinda similar happen with the rise of the Internet. I don’t know if you’ve been around with your friends when this happen, but the Internet has killed the pointless argument.

Pamela: Yes, Yes. [Laughter]

Fraser: So you’ll sit around with your friends and be like I think it’s this…no, no I think the first thing on Star Trek was that. Then someone will go well, I’ll just go look it up. And so, before the argument gets heated someone has already hit Wikipedia and has gotten the answer and conversation moves on. I think that this is kind of the same model is happening with science. Without science, without the scientific method people would have these philosophical discussions on how many angels dance on the head of a pin and, that’s a bad example. Because how many uh… [Laughter]

Pamela: That was a bit hard to experiment with.

Fraser: Yeah, exactly. That is a hard experiment. But you know what I’m getting at. Someone can say, well I can go out and measure them. How tall is a horse, well I’ll just go measure the horse and find out. You don’t need to even have an argument you just perform the measurement and get the answer.
 
I think that, that it’s that process at the very heart is just kind of saying, I don’t know anything. I’ll ask nature and nature will give me the answer. And once I’ve gotten the answer from nature, feel free to ask nature on your own as well. And, she should give you the same answer, right?

Pamela: And that’s the nice thing about a good theory. It’s completely reproducible. What I do in my laboratory, the observations I make with my telescope should be identical to the experiment that you do in your home that you do with your telescope.
 
Now, it gets a bit tricky with astronomy because say you’re the first person to observe some transitory event that occurs for 5 minutes. And you have no way of predicting when it’s going to happen next.
 
Well, then you sort of go well, I think I know what happened, I think I know what it was, I think my data was valid, but let’s hope we see this again. And, until someone is lucky enough to make the same observation we all sort of sit around and say, hmm, we really don’t know but it’s cool.

Fraser: Right. You, it’s not like you can go an explode stars until you think you’ve figured out how supernova works. You’ve gotta actually wait until one explodes. We don’t know when that’s going to happen. So I think astronomy is definitely different from a lot of the other sciences.

Pamela: We do a lot of sitting around. We think we’re right. But at the end of the day we have to wait for the universe to repeat our experiment for us.

Fraser: Okay, now I want to talk a bit about truth. And, sort of like, the theory of gravity you know, you call it the theory of gravity, you don’t call it the truth of gravity. [Laughter] Gravity is a theory. So is gravity still a theory?

Pamela: Well the word theory is something you have to be very careful using. It means different things to different people, which is always, always a terrible thing to happen when it happens to science.
 
There’s theory in terms of, oh God, he’s a theorist. You’re talking about someone who sits around and makes up mathematical models of the Universe that hurt to try and understand, and aren’t necessarily testable.
 
These are the extreme theorists out there. They’re wonderful people. I don’t understand what they do most of the time.

Fraser: Right. Like string theory, right? So far there’s no test devised that can actually test string theory.

Pamela: Right.

Fraser: The math might work but we’re still waiting for the test.

Pamela: And then there’s theory’s little ‘t’ which are things like well, I think and there are some theorists out there who agree with me that the Blazhko effect in RR Lyrae stars could be caused by magnetic fields but we don’t have the evidence yet to prove it or not.
 
So this is a theory, little ‘t’ which has yet to be tested that matches at least in the computer simulations what we think should happen. But it’s not proven. And then there’s the theory’s capital ‘T’.
 
These are rigorous mathematical models that have been extremely well tested, looked at upside down, inside out, matter, anti-matter, every way we can look at them. We can match what we see. We can match all the experiments we throw at them.
 
Or at least they have places that we can well define that they break. Like gravity it breaks inside of black holes. We know that. But in all these other well-defined regimes, no matter how we tested gravity as explained by Einstein and successors working on his theory of relativity it works. That’s a theory with a capital ‘T’.

Fraser: Right, evolution, Big Bang, a lot of these theories that have so many lines of evidence that I guess a lot of scientists consider them to be true. But, they’re still like all scientific theories are open for interpretation at some point.
 
Somebody could come along theoretically and come up with a new theory that explains all the evidence and pushes out further and you might have to throw some of those in the garbage. But it’s not very likely at this point.

Pamela: No it’s not very likely. That’s one of the cool things about the more advanced really well tested areas of science. We have this amazing intellectual tool set that we can apply to all the corners of the Universe that we’ve tested within our own laboratory planet very well. We can actually say solid things about how our universe is going to evolve based on the observations we can make in this one brief moment in time. And that’s just really cool.

Fraser: Now why do you think this scientific method is so powerful compared to other ways of knowing?

Pamela: Well, just the fact that it’s the ultimate here, let me show you why I’m right. It’s, you see two people having an argument, no, it’s red, no, it’s blue. Well, let me go get my meter, measure the wavelength of light coming off of it and show you exactly what color it is. Then you can argue over what adjective to use, but I can tell you a concrete number value, a concrete experimental value to end your argument.
 
There are lots of times when you hear people getting into arguments and one person shouts at the other: well, how do you know that? Well, scientific method allows me to say I know this because… and to list off the lines of evidence.
 
How do you know there is a Big Bang? Well, let me tell you about the helium abundances, let me tell you about the cosmic microwave background radiation. Let me tell you about lithium.
 
Let me tell you about so many different things that let me believe and let me demonstrate through the scientific method that this is something you should add to your lexicon as well. The Big Bang is real.

Fraser: And I think if a scientist has disconnected themselves from some kind of ego attachment to their ideas, they’re perfect happy to be proven wrong. You know, it’s like…I believe that you know it’s red and here is the evidence that I have that shows that I think it is red. If you can show me that it’s blue and show me better evidence that explains what I see, then you’re right.

Pamela: But we still cry. I actually had this happen with my dissertation. I had this theory, a perfectly good theory, that you could find galaxy clusters more easily by looking at over densities of radio galaxies in the sky. If you look out at the sky with a radio telescope and you find yourself a nice friendly galaxy that is giving off radio light, you have a 22 or 23 percent chance of finding several hundred galaxies clustered up around, or at least several tens of galaxies clustered up around that radio galaxy.
 
What I said was well, if you have this really good chance of finding a cluster, if you see one radio galaxy all by its lonesome, if you look at a patch of sky that has six radio galaxies all bunched up together, you increase your chance of finding a galaxy cluster. And I figure you increase it to like 60 or 70 percent, something really useful. No, you increase it by 3 percent to about 26 or 27 percent.
 
I spent a whole lot of time, a whole lot of observatory time, a whole lot of effort and six and a half years of my life increasing the probability of finding a galaxy cluster: 3 percent. I was very sad. I did cry. But it was the scientific method and sometimes you get results you don’t like and you have to go yeah, I tried.
 
Here are my results. I’m not exactly proud of how many years of my life I spent working on this but let’s all just move on together and don’t anyone else ever try this. It was a bad idea. But that’s the scientific method. You get sad you move on and you cling to your data and you acknowledge that it’s right even if you’re not.

Fraser: Right, right. You’re willing to accept that you’re wrong. At the end of the day the data is the final arbiter. That is what is truly happening in the Universe. What you hope and believe and what you feel in your gut is worthless.
 
I think that your instincts and your intuition about something that is happening (there is a bit of that) That helps you lead down areas of investigation and say well I wonder if this is what’s going on, or if I wonder if that’s what’s going on. Or hey, maybe it’s that cord that’s busted.
 
Until you actually perform the tests and do it well, you’re not able to come up with an answer. Now, I know Phil dealt with this recently. Why is science not faith? Why is the scientific method, belief in the scientific method not faith, or different from faith?

Pamela: With faith there is a certain element of I don’t have to understand what’s going on, I just have to accept it. With faith there is a certain amount of letting go and trusting in things that you can never understand.
 
With the scientific method and with science in general you may at a certain level have to let go and say I personally don’t have the capacity to understand this at this moment in time. Were I to sit down and let someone teach me I could understand why this is true within the region of experimentation that has already taken place.
 
With science, you don’t have to trust in a greater faith. You just may have to invest years of your life. Instead of investing those years of your life, you make the choice to say I’m going to trust these other people who have made that investment of time and education.
 
Science is straight-forward observation, mathematical rigor. Everything that we say scientifically we have a root in I saw this. We have a root in this is something that has been done and is repeatable. With faith, when you have a miracle, a miracle is something that can’t be explained and may only happen once and can never be repeated. You have to have faith that it wasn’t just some weird mental I’m going to see things that don’t actually exist.
 
With science, if you have a one off, it may be true, I don’t know. Let’s wait for the Universe to give us another Super Nova. Let’s wait for enough funding to repeat this laser pulse experiment.
 
Until you have that second set of data you don’t say that this is a scientifically valid result, you say this is a hint of something that might be scientifically valid given more data.
 
It’s that rigor, it’s the fact that we do say please do doubt. Please do question. Please test. Please force this to have a rigor that anyone can see with their eyes if they choose to invest the years of their life getting the necessary education. This is something that is true.

Fraser: What role do you think skepticism plays in this?

Pamela: Well, at the end of the day it’s easy to your first time through draw five things together and get an answer that does fit the data but fits it awkwardly, fits it poorly, it’s an ad hoc solution. With skepticism someone can come along and say dude, yeah, you put these pieces together wrong. Take them apart and let’s try this again.

Fraser: Well, you can even say just say show me, right? You say that this is true. Show me that evidence. Show me what you did. Let me test it I guess right?

Pamela: Yeah. This is one of the things that come up in the court system on a regular basis. You have one set of data brought forward and interpreted by one set of lawyers to show their case, one set of what they call truth.
 
Then another lawyer gets forward with another set of experts looks, at the exact set of data and comes up with a completely different set of results. It’s through the one lawyer questioning the other person’s experts, and the other lawyer questioning the other person’s set of experts, that this skepticism emerges.
 
Hopefully through the questioning process we can get to the truth. Now in reality the court system is an imperfect system where charisma and emotion play as important a role as science. But if we were able to set aside that emotion and simply work skeptically, simply question and seek truth as the data being the final authority versus what we want to be true to be the final authority.
 
It’s the data we need to trust in. Being skeptical is a way of saying I’m going to set my emotions aside. I’m going to be rigorous in my analysis even if my first results told me what I hoped they would say. I’m going to check my second and third results as well.

Fraser: I think that by nature, people are often skeptical about things that they don’t necessarily understand. Or think that they understand. But it’s almost like when you get to know something too well, you start to lose that skepticism.
 
I think a lot of the major mistakes in science have happened when people, scientists, have gotten too emotionally attached; their ideas and they’ve lost that skepticism. They’ve lost the chance to look at the data and just say what it is really saying. Not, is it saying what I want it to say. You know?

Pamela: There is also the stupid effect that we look at who is presenting the results. Sometimes we think they’re young, they can’t be right. Or oh they’re, and our lesser angels do the talking. Oh, they’re from that school I don’t like. Oh, their advisor is this person who ticked me off. Oh, this is that person who referred my paper and was bad to me. So you don’t listen to them, you don’t take them seriously.
 
Because of this several of the premiere results that have come out of astronomy have taken a long time to get accepted. Chandrasekhar came up with a model of a white dwarf on the boat to go to graduate school. It wasn’t until many years after he got his PhD that it was fully accepted and he eventually got a Nobel Prize for it.
 
For him it was the problem of he was putting forward theories that contradicted the rich famous dudes. He was this young Indian graduate student versus the expected scientist who has had the look and the heritage that was anticipated by people who had the key results. We have to set aside our emotional biases when we listen to the people presenting results.

Fraser: So, it’s almost like in a perfect world a lot of that stuff I guess would be considered, but wouldn’t take on the level of authority that it does and wouldn’t necessarily keep down new ideas as long as it does. I mean this is always an argument that is used against science. People never listened to Galileo. You know, and then…

Pamela: [Laughter] The thing with Galileo is the guy. I can’t say the words that described him best on the show because we’re safe for school kids. He was not a friendly dude and he stuck his foot so far down his mouth so many times.

Fraser: Yeah…but he was right.

Pamela: He was right. And when you’re really brilliant you have to be nice until people notice you’re brilliant. Then unfortunately a lot of really brilliant people become just not nice anymore.

Fraser: Yeah, they can be a jerk, right?

Pamela: Right. But Galileo didn’t wait until he had everyone convinced he was brilliant and right to start being not nice.

Fraser: Right. So, anyone working on science you know, take that lesson, always be nice.
 

Pamela: It always helps get you listened to.

Fraser: Well I hope this was able to answer people’s questions about the scientific method. I really think if there is one concept to really hammer home it’s this one.
 
If you’re interested in science at all, passing interest whatever, take some time to really understand how science works. Because then you can look at everything science does and really get a sense of where things are valid and where they’re not and how things move, and really appreciate the modern world and how things are changing around you.
 
I hope this helps and I hope this begins a discovery that you may take. Well thank you very much for teaching us again, Pamela.

Pamela: Oh, it’s been my pleasure. And all of you go out and use the scientific method in everything you do.

Fraser: But at least figure out why my phone doesn’t work.

Pamela: Exactly, exactly.

Fraser: We’ll talk to you next week.